2024-09-17T12:07:01.404 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T12:07:01.426 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T12:07:01.521 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-16_20:08:15-orch-main-distro-default-smithi/7907834 branch: main description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7907834' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-16_20:08:15-orch-main-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - CEPHADM_AGENT_DOWN log-only-match: - CEPHADM_ sha1: 7498493279982dfe87f00616198a5967475f6169 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 7498493279982dfe87f00616198a5967475f6169 workunit: branch: main sha1: 2b698643d05f73e79c2d2888228947a84a4ac0bd owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 3649 sha1: 7498493279982dfe87f00616198a5967475f6169 sleep_before_teardown: 0 subset: 42/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_2b698643d05f73e79c2d2888228947a84a4ac0bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 2b698643d05f73e79c2d2888228947a84a4ac0bd targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsCWX3h4hB49oNq62rIByJCJO/BLEncF/2tKULQEibNm/LEIqytR44dFLzMx/DV98I0l+7TaOxQn0B0+80xb5I= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEDTa0tOhMOO79q2cGVF7pCKxXPoB8Cl1cp4OQnFbj6SVzbKDcMOFqqvA4LX/KlBCWJZ0/1KwIRlI7Pk1Z1EdE= tasks: - cephadm: compiled_cephadm_branch: reef image: quay.ceph.io/ceph-ci/ceph:reef - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_20:08:15 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T12:07:01.521 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_2b698643d05f73e79c2d2888228947a84a4ac0bd/qa; will attempt to use it 2024-09-17T12:07:01.524 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_2b698643d05f73e79c2d2888228947a84a4ac0bd/qa/tasks 2024-09-17T12:07:01.524 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T12:07:01.526 INFO:teuthology.task.internal:Checking packages... 2024-09-17T12:07:01.547 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7498493279982dfe87f00616198a5967475f6169' 2024-09-17T12:07:01.548 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T12:07:01.548 INFO:teuthology.packaging:ref: None 2024-09-17T12:07:01.548 INFO:teuthology.packaging:tag: None 2024-09-17T12:07:01.548 INFO:teuthology.packaging:branch: main 2024-09-17T12:07:01.548 INFO:teuthology.packaging:sha1: 7498493279982dfe87f00616198a5967475f6169 2024-09-17T12:07:01.548 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-09-17T12:07:01.770 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4959.g2b698643 2024-09-17T12:07:01.772 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T12:07:01.778 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T12:07:01.778 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T12:07:01.784 INFO:teuthology.task.internal:Saving configuration 2024-09-17T12:07:01.797 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T12:07:01.803 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T12:07:01.828 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-16_20:08:15-orch-main-distro-default-smithi/7907834', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 12:01:57.679925', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsCWX3h4hB49oNq62rIByJCJO/BLEncF/2tKULQEibNm/LEIqytR44dFLzMx/DV98I0l+7TaOxQn0B0+80xb5I='} 2024-09-17T12:07:01.852 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-16_20:08:15-orch-main-distro-default-smithi/7907834', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 12:01:57.678053', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEDTa0tOhMOO79q2cGVF7pCKxXPoB8Cl1cp4OQnFbj6SVzbKDcMOFqqvA4LX/KlBCWJZ0/1KwIRlI7Pk1Z1EdE='} 2024-09-17T12:07:01.852 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T12:07:01.858 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-17T12:07:01.858 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-17T12:07:01.858 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T12:07:01.904 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5346ecbac0>, signals=[15]) 2024-09-17T12:07:01.904 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T12:07:01.910 INFO:teuthology.task.internal:Opening connections... 2024-09-17T12:07:01.911 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-09-17T12:07:01.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T12:07:01.987 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-09-17T12:07:01.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T12:07:02.064 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T12:07:02.071 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-09-17T12:07:02.089 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-09-17T12:07:02.090 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-09-17T12:07:02.147 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-09-17T12:07:02.147 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-09-17T12:07:02.147 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-09-17T12:07:02.147 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-09-17T12:07:02.147 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-09-17T12:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-09-17T12:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T12:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-09-17T12:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-09-17T12:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T12:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-09-17T12:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T12:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T12:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T12:07:02.149 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-09-17T12:07:02.179 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-09-17T12:07:02.197 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-09-17T12:07:02.198 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-09-17T12:07:02.255 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-09-17T12:07:02.255 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-09-17T12:07:02.255 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-09-17T12:07:02.255 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-09-17T12:07:02.255 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-09-17T12:07:02.256 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-09-17T12:07:02.256 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T12:07:02.256 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-09-17T12:07:02.256 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-09-17T12:07:02.256 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T12:07:02.256 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-09-17T12:07:02.256 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T12:07:02.256 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T12:07:02.256 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T12:07:02.257 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-09-17T12:07:02.279 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T12:07:02.288 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T12:07:02.294 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T12:07:02.294 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-09-17T12:07:02.297 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-09-17T12:07:02.313 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T12:07:02.320 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T12:07:02.320 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-09-17T12:07:02.355 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-09-17T12:07:02.380 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T12:07:02.423 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T12:07:02.438 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T12:07:02.439 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T12:07:02.439 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T12:07:02.439 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T12:07:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T12:07:02.456 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-09-17T12:07:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T12:07:02.457 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-17T12:07:02.513 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-503.el9.x86_64 2024-09-17T12:07:02.513 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-503.el9.x86_64 2024-09-17T12:07:02.513 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-09-17T12:07:02.514 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-503.el9.x86_64 2024-09-17T12:07:02.515 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-503.el9.x86_64 2024-09-17T12:07:02.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-09-17T12:07:05.286 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.3 MB 00:00 2024-09-17T12:07:07.312 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-17T12:07:07.787 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.3 MB 00:03 2024-09-17T12:07:09.521 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-17T12:07:12.721 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.5 MB 00:01 2024-09-17T12:07:14.214 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 19 kB 00:00 2024-09-17T12:07:14.272 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 6.5 MB 00:00 2024-09-17T12:07:15.044 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 32 MB/s | 23 MB 00:00 2024-09-17T12:07:15.814 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 94 kB/s | 19 kB 00:00 2024-09-17T12:07:16.593 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-17T12:07:20.632 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-17T12:07:21.924 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T12:07:22.597 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T12:07:22.598 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T12:07:22.693 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-17T12:07:22.703 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-17T12:07:22.703 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-09-17T12:07:22.703 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-17T12:07:22.703 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-17T12:07:22.703 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T12:07:22.703 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-17T12:07:22.703 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T12:07:22.703 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T12:07:22.703 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T12:07:22.703 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:07:22.704 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-17T12:07:22.704 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-17T12:07:22.704 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-09-17T12:07:22.704 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:07:22.704 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 79 M 2024-09-17T12:07:22.704 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 126 M 2024-09-17T12:07:22.704 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-17T12:07:22.933 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 277 kB/s | 36 kB 00:00 2024-09-17T12:07:23.855 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T12:07:23.856 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T12:07:23.945 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-17T12:07:23.951 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-17T12:07:23.951 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-09-17T12:07:23.951 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-17T12:07:23.951 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-17T12:07:23.951 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T12:07:23.951 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-17T12:07:23.951 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T12:07:23.951 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T12:07:23.952 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T12:07:23.952 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-17T12:07:23.952 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-17T12:07:23.952 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-17T12:07:23.952 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-09-17T12:07:23.952 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-17T12:07:23.952 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 79 M 2024-09-17T12:07:23.953 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 126 M 2024-09-17T12:07:23.953 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-17T12:07:24.165 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 342 kB/s | 36 kB 00:00 2024-09-17T12:07:24.709 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-09-17T12:07:25.259 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 26 MB/s | 29 MB 00:01 2024-09-17T12:07:25.601 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 22 MB/s | 35 MB 00:01 2024-09-17T12:07:25.601 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-17T12:07:25.601 INFO:teuthology.orchestra.run.smithi196.stdout:Total 48 MB/s | 79 MB 00:01 2024-09-17T12:07:26.281 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-17T12:07:26.629 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-17T12:07:26.629 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-17T12:07:27.125 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): kernel-modules-core-5.14.0-508.el9.x86_6 6.8 MB/s | 29 MB 00:04 2024-09-17T12:07:27.617 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-17T12:07:27.617 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-17T12:07:30.076 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-17T12:07:30.100 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 2.1 MB/s | 16 MB 00:07 2024-09-17T12:07:30.713 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T12:07:30.794 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T12:07:33.546 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T12:07:33.679 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T12:07:38.574 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T12:07:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T12:07:39.350 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 2.1 MB/s | 35 MB 00:16 2024-09-17T12:07:39.350 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-17T12:07:39.350 INFO:teuthology.orchestra.run.smithi016.stdout:Total 4.7 MB/s | 79 MB 00:16 2024-09-17T12:07:40.065 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-17T12:07:40.440 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-17T12:07:40.440 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-17T12:07:41.422 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-17T12:07:41.423 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-17T12:07:43.487 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T12:07:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-17T12:07:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T12:07:44.717 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T12:07:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T12:07:47.624 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T12:07:52.750 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T12:07:52.805 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T12:07:57.950 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T12:08:21.563 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T12:08:21.603 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T12:08:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T12:08:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T12:08:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T12:08:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T12:08:22.789 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T12:08:22.789 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-17T12:08:22.789 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-17T12:08:22.789 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T12:08:22.789 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T12:08:22.789 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T12:08:22.790 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T12:08:22.790 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-17T12:08:22.790 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-17T12:08:23.123 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T12:08:23.214 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T12:08:23.215 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T12:08:23.215 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T12:08:23.215 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T12:08:23.215 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T12:08:23.215 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-17T12:08:23.216 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T12:08:23.216 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T12:08:23.275 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T12:08:23.275 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T12:08:23.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-09-17T12:08:23.323 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-09-17T12:08:23.323 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-09-17T12:08:23.323 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-09-17T12:08:23.323 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 82.el9 2024-09-17T12:08:23.324 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-09-17T12:08:23.324 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T12:08:23.324 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-09-17T12:08:23.324 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8274670 2024-09-17T12:08:23.324 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-09-17T12:08:23.324 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T12:08:23.324 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T12:08:23.324 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T12:08:23.324 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T12:08:23.324 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-09-17T12:08:23.324 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-09-17T12:08:23.324 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T12:08:23.324 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-09-17T12:08:23.325 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-09-17T12:08:23.325 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-17T12:08:23.325 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T12:08:23.325 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T12:08:23.325 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T12:08:23.325 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-09-17T12:08:23.325 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-17T12:08:23.325 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T12:08:23.326 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T12:08:23.326 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-508.el9.x86_64 2024-09-17T12:08:23.326 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T12:08:23.965 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-09-17T12:08:25.261 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T12:08:25.311 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-09-17T12:08:25.313 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-09-17T12:08:25.328 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.RyynOd0f1t 2024-09-17T12:08:25.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RyynOd0f1t 2024-09-17T12:08:25.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.RyynOd0f1t 2024-09-17T12:08:25.534 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.RyynOd0f1t is 6KB 2024-09-17T12:08:25.582 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.RyynOd0f1t 2024-09-17T12:08:25.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T12:08:25.660 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T12:08:25.661 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T12:08:25.661 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T12:08:25.661 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T12:08:25.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T12:08:25.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-09-17T12:08:36.751 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T12:08:36.794 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T12:08:37.125 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T12:08:37.125 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T12:08:37.125 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T12:08:37.125 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T12:08:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T12:08:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:08:38.005 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-17T12:08:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T12:08:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T12:08:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T12:08:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T12:08:38.006 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:08:38.006 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-17T12:08:38.371 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T12:08:38.470 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T12:08:38.470 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T12:08:38.470 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T12:08:38.471 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T12:08:38.471 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T12:08:38.471 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T12:08:38.471 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T12:08:38.471 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T12:08:38.529 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T12:08:38.530 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T12:08:38.530 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2024-09-17T12:08:38.563 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2024-09-17T12:08:38.563 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2024-09-17T12:08:38.563 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.06 2024-09-17T12:08:38.563 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 82.el9 2024-09-17T12:08:38.563 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2024-09-17T12:08:38.563 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T12:08:38.563 INFO:teuthology.orchestra.run.smithi016.stdout:Group : Unspecified 2024-09-17T12:08:38.563 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 8274670 2024-09-17T12:08:38.564 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2024-09-17T12:08:38.564 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T12:08:38.564 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T12:08:38.564 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T12:08:38.564 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T12:08:38.564 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : builder@centos.org 2024-09-17T12:08:38.564 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : CentOS 2024-09-17T12:08:38.564 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T12:08:38.564 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2024-09-17T12:08:38.564 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2024-09-17T12:08:38.564 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:08:38.564 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T12:08:38.564 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T12:08:38.565 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T12:08:38.565 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2024-09-17T12:08:38.565 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:08:38.565 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T12:08:38.566 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T12:08:38.566 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 5.14.0-508.el9.x86_64 2024-09-17T12:08:38.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T12:08:39.230 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-09-17T12:08:40.475 INFO:teuthology.orchestra.run.smithi016.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T12:08:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-09-17T12:08:40.518 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-17T12:08:40.541 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.dCYSmWMcUF 2024-09-17T12:08:40.542 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dCYSmWMcUF 2024-09-17T12:08:40.607 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.dCYSmWMcUF 2024-09-17T12:08:40.747 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.dCYSmWMcUF is 6KB 2024-09-17T12:08:40.795 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.dCYSmWMcUF 2024-09-17T12:08:40.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T12:08:40.873 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T12:08:40.874 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T12:08:40.874 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T12:08:40.874 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T12:08:40.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T12:08:41.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-09-17T12:08:55.845 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-17T12:08:55.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-17T12:08:55.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T12:09:11.041 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T12:09:11.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-17T12:09:11.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T12:09:14.301 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-09-17T12:09:23.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-17T12:09:23.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T12:09:26.365 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-09-17T12:09:29.405 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-09-17T12:09:38.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-17T12:09:38.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T12:09:38.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-17T12:09:38.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T12:09:41.437 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-09-17T12:09:41.469 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-09-17T12:09:53.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-17T12:09:53.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T12:09:56.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-17T12:09:56.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T12:09:59.521 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-09-17T12:10:17.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-17T12:10:17.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T12:10:17.985 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-09-17T12:10:18.343 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-17T12:10:18.343 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-17T12:10:18.344 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-09-17T12:10:18.360 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-508.el9.x86_64 2024-09-17T12:10:18.360 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T12:10:18.361 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T12:10:18.361 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T12:10:19.362 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-17T12:10:19.363 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T12:10:19.435 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-09-17T12:10:19.457 DEBUG:teuthology.parallel:result is None 2024-09-17T12:10:25.278 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-17T12:10:25.899 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-17T12:10:25.900 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T12:10:25.900 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-17T12:10:25.916 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-508.el9.x86_64 2024-09-17T12:10:25.917 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T12:10:25.917 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T12:10:25.917 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T12:10:26.918 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T12:10:26.919 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T12:10:26.983 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-09-17T12:10:27.004 DEBUG:teuthology.parallel:result is None 2024-09-17T12:10:27.005 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T12:10:27.012 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T12:10:27.012 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T12:10:27.015 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T12:10:27.034 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T12:10:27.040 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T12:10:27.048 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T12:10:27.048 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T12:10:27.072 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T12:10:27.100 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T12:10:27.106 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T12:10:27.106 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T12:10:27.137 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T12:10:27.164 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T12:10:27.176 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T12:10:27.182 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T12:10:27.194 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T12:10:27.196 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T12:10:27.203 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T12:10:27.203 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T12:10:27.219 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T12:10:27.248 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T12:10:27.257 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T12:10:27.257 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T12:10:27.285 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T12:10:27.305 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T12:10:27.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T12:10:27.435 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T12:10:27.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T12:10:27.558 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-17T12:10:27.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T12:10:27.623 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T12:10:27.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T12:10:27.719 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T12:10:27.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T12:10:27.845 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-17T12:10:27.845 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T12:10:27.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-09-17T12:10:27.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-09-17T12:10:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T12:10:28.002 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T12:10:28.468 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T12:10:28.476 INFO:teuthology.task.internal:Starting timer... 2024-09-17T12:10:28.476 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T12:10:28.490 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T12:10:28.499 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-09-17T12:10:28.585 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-09-17T12:10:28.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-09-17T12:10:28.644 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-09-17T12:10:28.646 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T12:10:28.646 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-09-17T12:10:28.672 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-09-17T12:10:28.672 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-09-17T12:10:28.704 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-09-17T12:10:28.704 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-09-17T12:10:28.705 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T12:10:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T12:10:28.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T12:10:28.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T12:10:28.774 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T12:10:28.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-09-17T12:10:28.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-09-17T12:10:28.845 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T12:10:28.855 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-17T12:10:28.858 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-17T12:10:28.970 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T12:10:28.985 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T12:10:28.987 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T12:14:15.374 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-09-17T12:14:15.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-17T12:14:15.377 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T12:14:15.458 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-17T12:14:15.541 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-17T12:14:15.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-17T12:14:15.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T12:14:15.616 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-09-17T12:14:15.692 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-17T12:14:15.692 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T12:14:15.703 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T12:14:15.703 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T12:14:15.704 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T12:14:15.706 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T12:14:15.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T12:14:15.735 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T12:14:15.751 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T12:14:15.760 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T12:14:15.774 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T12:14:15.777 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-09-17T12:14:15.789 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-09-17T12:14:15.799 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-09-17T12:14:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T12:14:15.810 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-09-17T12:14:15.818 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T12:14:15.824 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T12:14:15.838 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T12:14:15.876 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-09-17T12:14:15.880 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T12:14:15.881 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-09-17T12:14:15.881 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T12:14:15.881 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T12:14:15.881 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T12:14:15.881 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T12:14:15.897 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-09-17T12:14:15.900 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T12:14:15.900 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-09-17T12:14:15.901 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T12:14:15.901 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T12:14:15.901 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T12:14:15.901 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T12:14:15.905 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-17T12:14:15.997 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'CEPHADM_AGENT_DOWN'], 'log-only-match': ['CEPHADM_'], 'sha1': '7498493279982dfe87f00616198a5967475f6169'} 2024-09-17T12:14:15.997 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-09-17T12:14:15.997 INFO:tasks.cephadm:Cluster fsid is 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:14:15.998 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-17T12:14:15.998 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.16', 'mon.c': '[v2:172.21.15.16:3301,v1:172.21.15.16:6790]', 'mon.b': '172.21.15.196'} 2024-09-17T12:14:15.998 INFO:tasks.cephadm:First mon is mon.a on smithi016 2024-09-17T12:14:15.998 INFO:tasks.cephadm:First mgr is y 2024-09-17T12:14:15.998 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-17T12:14:15.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2024-09-17T12:14:16.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo hostname $(hostname -s) 2024-09-17T12:14:16.052 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-09-17T12:14:16.053 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7498493279982dfe87f00616198a5967475f6169 2024-09-17T12:14:16.199 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '7498493279982dfe87f00616198a5967475f6169', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56090/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4937-g74984932', 'node_name': '172.21.2.22+braggi22', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4937.g74984932'}, 'url': 'https://4.chacra.ceph.com/r/ceph/main/7498493279982dfe87f00616198a5967475f6169/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-15 22:16:53.267223', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/main/7498493279982dfe87f00616198a5967475f6169/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-17T12:14:16.485 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref reef, sha1 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-09-17T12:14:16.487 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/x86_64/flavors/default/cephadm 2024-09-17T12:14:16.487 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/x86_64/flavors/default/cephadm 2024-09-17T12:14:16.487 DEBUG:teuthology.orchestra.run.smithi016:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T12:14:16.848 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r--. 1 ubuntu ubuntu 217222 Sep 17 12:14 /home/ubuntu/cephtest/cephadm 2024-09-17T12:14:16.848 DEBUG:teuthology.orchestra.run.smithi196:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T12:14:17.171 INFO:teuthology.orchestra.run.smithi196.stdout:-rw-r--r--. 1 ubuntu ubuntu 217222 Sep 17 12:14 /home/ubuntu/cephtest/cephadm 2024-09-17T12:14:17.171 DEBUG:teuthology.orchestra.run.smithi016:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T12:14:17.191 DEBUG:teuthology.orchestra.run.smithi196:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T12:14:17.223 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-09-17T12:14:17.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-17T12:14:17.234 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-17T12:14:18.683 INFO:teuthology.orchestra.run.smithi196.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-17T12:14:18.960 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-17T12:14:49.111 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-17T12:14:49.112 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_version": "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)", 2024-09-17T12:14:49.112 INFO:teuthology.orchestra.run.smithi196.stdout: "image_id": "980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89", 2024-09-17T12:14:49.112 INFO:teuthology.orchestra.run.smithi196.stdout: "repo_digests": [ 2024-09-17T12:14:49.112 INFO:teuthology.orchestra.run.smithi196.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9" 2024-09-17T12:14:49.112 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-17T12:14:49.112 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-17T12:15:16.969 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:15:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)", 2024-09-17T12:15:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89", 2024-09-17T12:15:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2024-09-17T12:15:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9" 2024-09-17T12:15:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-17T12:15:16.970 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:15:16.989 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2024-09-17T12:15:17.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph 2024-09-17T12:15:17.073 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2024-09-17T12:15:17.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /etc/ceph 2024-09-17T12:15:17.141 INFO:tasks.cephadm:Writing seed config... 2024-09-17T12:15:17.142 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-17T12:15:17.143 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-17T12:15:17.143 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-17T12:15:17.143 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-09-17T12:15:17.143 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-17T12:15:17.143 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-17T12:15:17.143 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-17T12:15:17.143 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-17T12:15:17.143 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-17T12:15:17.144 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-17T12:15:17.144 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-17T12:15:17.161 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 5b9f4610-74ee-11ef-bceb-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-17T12:15:17.161 DEBUG:teuthology.orchestra.run.smithi016:mon.a> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.a.service 2024-09-17T12:15:17.203 DEBUG:teuthology.orchestra.run.smithi016:mgr.y> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.y.service 2024-09-17T12:15:17.246 INFO:tasks.cephadm:Bootstrapping... 2024-09-17T12:15:17.247 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.16 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-17T12:15:17.405 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-17T12:15:17.405 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', '5b9f4610-74ee-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.16', '--skip-admin-label'] 2024-09-17T12:15:17.429 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5.2.2 2024-09-17T12:15:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-17T12:15:17.430 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying podman|docker is present... 2024-09-17T12:15:17.471 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5.2.2 2024-09-17T12:15:17.471 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying lvm2 is present... 2024-09-17T12:15:17.471 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying time synchronization is in place... 2024-09-17T12:15:17.479 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-17T12:15:17.479 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T12:15:17.485 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-17T12:15:17.485 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-17T12:15:17.492 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-09-17T12:15:17.498 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-09-17T12:15:17.498 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-09-17T12:15:17.498 INFO:teuthology.orchestra.run.smithi016.stdout:Repeating the final host check... 2024-09-17T12:15:17.522 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5.2.2 2024-09-17T12:15:17.523 INFO:teuthology.orchestra.run.smithi016.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-17T12:15:17.523 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl is present 2024-09-17T12:15:17.523 INFO:teuthology.orchestra.run.smithi016.stdout:lvcreate is present 2024-09-17T12:15:17.529 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-17T12:15:17.529 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T12:15:17.536 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-17T12:15:17.536 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-17T12:15:17.542 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-09-17T12:15:17.549 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-09-17T12:15:17.549 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-09-17T12:15:17.549 INFO:teuthology.orchestra.run.smithi016.stdout:Host looks OK 2024-09-17T12:15:17.549 INFO:teuthology.orchestra.run.smithi016.stdout:Cluster fsid: 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:15:17.550 INFO:teuthology.orchestra.run.smithi016.stdout:Acquiring lock 140197326409888 on /run/cephadm/5b9f4610-74ee-11ef-bceb-c7b262605968.lock 2024-09-17T12:15:17.550 INFO:teuthology.orchestra.run.smithi016.stdout:Lock 140197326409888 acquired on /run/cephadm/5b9f4610-74ee-11ef-bceb-c7b262605968.lock 2024-09-17T12:15:17.550 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 3300 ... 2024-09-17T12:15:17.550 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 6789 ... 2024-09-17T12:15:17.550 INFO:teuthology.orchestra.run.smithi016.stdout:Base mon IP(s) is [172.21.15.16:3300, 172.21.15.16:6789], mon addrv is [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-09-17T12:15:17.553 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.16 metric 100 2024-09-17T12:15:17.553 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.16 metric 100 2024-09-17T12:15:17.556 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-17T12:15:17.556 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-17T12:15:17.556 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2024-09-17T12:15:17.559 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-17T12:15:17.559 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-17T12:15:17.559 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-17T12:15:17.559 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-17T12:15:17.559 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-09-17T12:15:17.559 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-17T12:15:17.560 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-09-17T12:15:17.560 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-09-17T12:15:17.560 INFO:teuthology.orchestra.run.smithi016.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-17T12:15:17.560 INFO:teuthology.orchestra.run.smithi016.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-17T12:15:17.560 INFO:teuthology.orchestra.run.smithi016.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-17T12:15:18.055 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 2024-09-17T12:15:18.055 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-09-17T12:15:18.055 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Getting image source signatures 2024-09-17T12:15:18.055 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:d8a707c21dc06a64315160cb2d8e524fa91943a344731b2a930ee3a716bebbdd 2024-09-17T12:15:18.055 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-17T12:15:18.055 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying config sha256:980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 2024-09-17T12:15:18.055 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-17T12:15:18.867 INFO:teuthology.orchestra.run.smithi016.stdout:ceph: stdout ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable) 2024-09-17T12:15:18.867 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph version: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable) 2024-09-17T12:15:18.868 INFO:teuthology.orchestra.run.smithi016.stdout:Extracting ceph user uid/gid from container image... 2024-09-17T12:15:19.634 INFO:teuthology.orchestra.run.smithi016.stdout:stat: stdout 167 167 2024-09-17T12:15:19.634 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial keys... 2024-09-17T12:15:20.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQDXculmO1T+OhAAGK/dD8b0+Ozi1u/VcxyYTw== 2024-09-17T12:15:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQDYculmzB/0NRAAjD+JlUZ0gpcPby1nIlyOZA== 2024-09-17T12:15:22.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQDZculmx7FlKhAASvvCyEC7BMLFxx9spiM7ng== 2024-09-17T12:15:22.411 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial monmap... 2024-09-17T12:15:23.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T12:15:23.242 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-09-17T12:15:23.242 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:15:23.242 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T12:15:23.242 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool for a [v2:172.21.15.16:3300,v1:172.21.15.16:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T12:15:23.242 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = pacific 2024-09-17T12:15:23.243 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: set fsid to 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:15:23.243 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T12:15:23.243 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:15:23.243 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mon... 2024-09-17T12:15:24.171 INFO:teuthology.orchestra.run.smithi016.stdout:create mon.a on 2024-09-17T12:15:24.550 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-17T12:15:24.761 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-5b9f4610-74ee-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-5b9f4610-74ee-11ef-bceb-c7b262605968.target. 2024-09-17T12:15:24.761 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-5b9f4610-74ee-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-5b9f4610-74ee-11ef-bceb-c7b262605968.target. 2024-09-17T12:15:25.007 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.a 2024-09-17T12:15:25.007 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-17T12:15:25.188 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5b9f4610-74ee-11ef-bceb-c7b262605968.target.wants/ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@.service. 2024-09-17T12:15:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T12:15:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-09-17T12:15:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T12:15:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-17T12:15:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-09-17T12:15:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-09-17T12:15:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon to start... 2024-09-17T12:15:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon... 2024-09-17T12:15:26.769 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout cluster: 2024-09-17T12:15:26.769 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout id: 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:15:26.769 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-17T12:15:26.769 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-17T12:15:26.769 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout services: 2024-09-17T12:15:26.770 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.529317s) 2024-09-17T12:15:26.770 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-17T12:15:26.770 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-17T12:15:26.770 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-17T12:15:26.770 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout data: 2024-09-17T12:15:26.770 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-17T12:15:26.770 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-17T12:15:26.770 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-17T12:15:26.770 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pgs: 2024-09-17T12:15:26.770 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-17T12:15:26.770 INFO:teuthology.orchestra.run.smithi016.stdout:mon is available 2024-09-17T12:15:26.770 INFO:teuthology.orchestra.run.smithi016.stdout:Assimilating anything we can from ceph.conf... 2024-09-17T12:15:27.700 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-17T12:15:27.700 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-09-17T12:15:27.700 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:15:27.700 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-09-17T12:15:27.700 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-17T12:15:27.700 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-17T12:15:27.700 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-17T12:15:27.700 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-17T12:15:27.700 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-17T12:15:27.700 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-17T12:15:27.700 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2024-09-17T12:15:27.700 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-17T12:15:27.701 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-17T12:15:27.701 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-09-17T12:15:27.701 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-17T12:15:27.701 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-17T12:15:27.701 INFO:teuthology.orchestra.run.smithi016.stdout:Generating new minimal ceph.conf... 2024-09-17T12:15:28.722 INFO:teuthology.orchestra.run.smithi016.stdout:Restarting the monitor... 2024-09-17T12:15:28.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:28 smithi016 systemd[1]: Stopping Ceph mon.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:15:29.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:28 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a[23341]: 2024-09-17T12:15:28.971+0000 7fd1666c4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:15:29.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:28 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a[23341]: 2024-09-17T12:15:28.971+0000 7fd1666c4640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-17T12:15:29.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:29 smithi016 podman[23589]: 2024-09-17 12:15:29.216411741 +0000 UTC m=+0.330393096 container died 51f1f882483d274f78b23941a2d6e42e205f4dcd2b09352130e97245d66213b8 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2) 2024-09-17T12:15:33.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:32 smithi016 podman[23589]: 2024-09-17 12:15:32.927579138 +0000 UTC m=+4.041560486 container cleanup 51f1f882483d274f78b23941a2d6e42e205f4dcd2b09352130e97245d66213b8 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-17T12:15:33.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:32 smithi016 bash[23589]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a 2024-09-17T12:15:35.238 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:34 smithi016 podman[23601]: 2024-09-17 12:15:34.946615728 +0000 UTC m=+5.729647491 container remove 51f1f882483d274f78b23941a2d6e42e205f4dcd2b09352130e97245d66213b8 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b) 2024-09-17T12:15:35.238 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:35 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-17T12:15:35.238 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:35 smithi016 systemd[1]: Stopped Ceph mon.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:15:35.238 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:35 smithi016 systemd[1]: Starting Ceph mon.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:15:35.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:35 smithi016 podman[23698]: 2024-09-17 12:15:35.191488077 +0000 UTC m=+0.016509744 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:reef 2024-09-17T12:15:36.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:35 smithi016 podman[23698]: 2024-09-17 12:15:35.802677775 +0000 UTC m=+0.627699432 container create 5d21f110d6fca319f092f5ad2a5980f2fbff43b6fbcc051eb597b4d37570e531 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:15:36.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 podman[23698]: 2024-09-17 12:15:36.01054607 +0000 UTC m=+0.835567728 container init 5d21f110d6fca319f092f5ad2a5980f2fbff43b6fbcc051eb597b4d37570e531 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-17T12:15:36.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 podman[23698]: 2024-09-17 12:15:36.013998121 +0000 UTC m=+0.839019779 container start 5d21f110d6fca319f092f5ad2a5980f2fbff43b6fbcc051eb597b4d37570e531 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-17T12:15:36.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T12:15:36.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process ceph-mon, pid 2 2024-09-17T12:15:36.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: pidfile_write: ignore empty --pid-file 2024-09-17T12:15:36.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: load: jerasure load: lrc 2024-09-17T12:15:36.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: RocksDB version: 7.9.2 2024-09-17T12:15:36.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Git sha 0 2024-09-17T12:15:36.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Compile date 2024-09-16 15:15:13 2024-09-17T12:15:36.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: DB SUMMARY 2024-09-17T12:15:36.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: DB Session ID: I308ALD94CCB7PJBV5RG 2024-09-17T12:15:36.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: CURRENT file: CURRENT 2024-09-17T12:15:36.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T12:15:36.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-17T12:15:36.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-17T12:15:36.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 83640 ; 2024-09-17T12:15:36.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.error_if_exists: 0 2024-09-17T12:15:36.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.create_if_missing: 0 2024-09-17T12:15:36.089 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.paranoid_checks: 1 2024-09-17T12:15:36.089 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T12:15:36.089 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T12:15:36.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T12:15:36.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.env: 0x5581d7fadbc0 2024-09-17T12:15:36.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T12:15:36.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.info_log: 0x5581d9816be0 2024-09-17T12:15:36.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T12:15:36.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.statistics: (nil) 2024-09-17T12:15:36.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.use_fsync: 0 2024-09-17T12:15:36.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_log_file_size: 0 2024-09-17T12:15:36.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T12:15:36.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T12:15:36.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T12:15:36.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T12:15:36.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.allow_fallocate: 1 2024-09-17T12:15:36.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T12:15:36.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T12:15:36.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.use_direct_reads: 0 2024-09-17T12:15:36.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T12:15:36.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T12:15:36.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.db_log_dir: 2024-09-17T12:15:36.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.wal_dir: 2024-09-17T12:15:36.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T12:15:36.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T12:15:36.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T12:15:36.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T12:15:36.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T12:15:36.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T12:15:36.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T12:15:36.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T12:15:36.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.write_buffer_manager: 0x5581d9824320 2024-09-17T12:15:36.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T12:15:36.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T12:15:36.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T12:15:36.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T12:15:36.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T12:15:36.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T12:15:36.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T12:15:36.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T12:15:36.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.unordered_write: 0 2024-09-17T12:15:36.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T12:15:36.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T12:15:36.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T12:15:36.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T12:15:36.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.row_cache: None 2024-09-17T12:15:36.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.wal_filter: None 2024-09-17T12:15:36.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T12:15:36.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T12:15:36.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.two_write_queues: 0 2024-09-17T12:15:36.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T12:15:36.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.wal_compression: 0 2024-09-17T12:15:36.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.atomic_flush: 0 2024-09-17T12:15:36.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T12:15:36.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T12:15:36.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T12:15:36.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.log_readahead_size: 0 2024-09-17T12:15:36.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T12:15:36.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T12:15:36.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T12:15:36.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T12:15:36.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T12:15:36.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T12:15:36.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T12:15:36.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_background_jobs: 2 2024-09-17T12:15:36.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_background_compactions: -1 2024-09-17T12:15:36.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_subcompactions: 1 2024-09-17T12:15:36.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T12:15:36.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T12:15:36.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T12:15:36.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T12:15:36.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T12:15:36.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T12:15:36.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T12:15:36.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T12:15:36.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_open_files: -1 2024-09-17T12:15:36.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T12:15:36.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T12:15:36.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T12:15:36.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T12:15:36.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_background_flushes: -1 2024-09-17T12:15:36.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Compression algorithms supported: 2024-09-17T12:15:36.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: kZSTD supported: 0 2024-09-17T12:15:36.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: kXpressCompression supported: 0 2024-09-17T12:15:36.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: kBZip2Compression supported: 0 2024-09-17T12:15:36.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T12:15:36.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: kLZ4Compression supported: 1 2024-09-17T12:15:36.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: kZlibCompression supported: 1 2024-09-17T12:15:36.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T12:15:36.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: kSnappyCompression supported: 1 2024-09-17T12:15:36.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T12:15:36.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T12:15:36.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-17T12:15:36.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T12:15:36.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T12:15:36.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.merge_operator: 2024-09-17T12:15:36.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compaction_filter: None 2024-09-17T12:15:36.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T12:15:36.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T12:15:36.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5581d98167e0) 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_type: 0 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_index_type: 0 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_shortening: 1 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout: checksum: 4 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout: no_block_cache: 0 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache: 0x5581d980b350 2024-09-17T12:15:36.098 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-09-17T12:15:36.099 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_options: 2024-09-17T12:15:36.099 INFO:journalctl@ceph.mon.a.smithi016.stdout: capacity : 536870912 2024-09-17T12:15:36.099 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_shard_bits : 4 2024-09-17T12:15:36.099 INFO:journalctl@ceph.mon.a.smithi016.stdout: strict_capacity_limit : 0 2024-09-17T12:15:36.099 INFO:journalctl@ceph.mon.a.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-09-17T12:15:36.099 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_compressed: (nil) 2024-09-17T12:15:36.099 INFO:journalctl@ceph.mon.a.smithi016.stdout: persistent_cache: (nil) 2024-09-17T12:15:36.099 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size: 4096 2024-09-17T12:15:36.099 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size_deviation: 10 2024-09-17T12:15:36.099 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_restart_interval: 16 2024-09-17T12:15:36.099 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_block_restart_interval: 1 2024-09-17T12:15:36.099 INFO:journalctl@ceph.mon.a.smithi016.stdout: metadata_block_size: 4096 2024-09-17T12:15:36.099 INFO:journalctl@ceph.mon.a.smithi016.stdout: partition_filters: 0 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout: use_delta_encoding: 1 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout: filter_policy: bloomfilter 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout: whole_key_filtering: 1 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout: verify_compression: 0 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout: format_version: 5 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_index_compression: 1 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_align: 0 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout: max_auto_readahead_size: 262144 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout: prepopulate_block_cache: 0 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout: initial_auto_readahead_size: 8192 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T12:15:36.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compression: NoCompression 2024-09-17T12:15:36.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T12:15:36.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T12:15:36.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T12:15:36.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.num_levels: 7 2024-09-17T12:15:36.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T12:15:36.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T12:15:36.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T12:15:36.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T12:15:36.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T12:15:36.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T12:15:36.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T12:15:36.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T12:15:36.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T12:15:36.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T12:15:36.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T12:15:36.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T12:15:36.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T12:15:36.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T12:15:36.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T12:15:36.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T12:15:36.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T12:15:36.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T12:15:36.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T12:15:36.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T12:15:36.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T12:15:36.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T12:15:36.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T12:15:36.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T12:15:36.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T12:15:36.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T12:15:36.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T12:15:36.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T12:15:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T12:15:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T12:15:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T12:15:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T12:15:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T12:15:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T12:15:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T12:15:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T12:15:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T12:15:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T12:15:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T12:15:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T12:15:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T12:15:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T12:15:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T12:15:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T12:15:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T12:15:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T12:15:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T12:15:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T12:15:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T12:15:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T12:15:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T12:15:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T12:15:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T12:15:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T12:15:36.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.inplace_update_support: 0 2024-09-17T12:15:36.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T12:15:36.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T12:15:36.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T12:15:36.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T12:15:36.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.bloom_locality: 0 2024-09-17T12:15:36.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.max_successive_merges: 0 2024-09-17T12:15:36.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T12:15:36.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T12:15:36.106 INFO:teuthology.orchestra.run.smithi016.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-17T12:15:36.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T12:15:36.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T12:15:36.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.ttl: 2592000 2024-09-17T12:15:36.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T12:15:36.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T12:15:36.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.enable_blob_files: false 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.min_blob_size: 0 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a064515c-6d2f-4b13-9e9e-c1520d5f10d2 2024-09-17T12:15:36.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726575336041673, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-17T12:15:36.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-17T12:15:36.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726575336042611, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 80289, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 78450, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 72761, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726575336, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a064515c-6d2f-4b13-9e9e-c1520d5f10d2", "db_session_id": "I308ALD94CCB7PJBV5RG", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-17T12:15:36.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726575336042688, "job": 1, "event": "recovery_finished"} 2024-09-17T12:15:36.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-17T12:15:36.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T12:15:36.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5581d9836e00 2024-09-17T12:15:36.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: DB pointer 0x5581d992a000 2024-09-17T12:15:36.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T12:15:36.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T12:15:36.109 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** DB Stats ** 2024-09-17T12:15:36.109 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T12:15:36.109 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: L0 2/0 80.27 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: Sum 2/0 80.27 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-17T12:15:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-09-17T12:15:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T12:15:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T12:15:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 126.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T12:15:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-17T12:15:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T12:15:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-17T12:15:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T12:15:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T12:15:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T12:15:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T12:15:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T12:15:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T12:15:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative compaction: 0.00 GB write, 24.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T12:15:36.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval compaction: 0.00 GB write, 24.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T12:15:36.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T12:15:36.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache BinnedLRUCache@0x5581d980b350#2 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-17T12:15:36.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-09-17T12:15:36.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-17T12:15:36.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T12:15:36.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: starting mon.a rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:15:36.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: mon.a@-1(???) e1 preinit fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:15:36.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: mon.a@-1(???).mds e1 new map 2024-09-17T12:15:36.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: mon.a@-1(???).mds e1 print_map 2024-09-17T12:15:36.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: e1 2024-09-17T12:15:36.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T12:15:36.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T12:15:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: legacy client fscid: -1 2024-09-17T12:15:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-17T12:15:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: No filesystems configured 2024-09-17T12:15:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T12:15:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T12:15:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T12:15:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T12:15:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-17T12:15:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 bash[23698]: 5d21f110d6fca319f092f5ad2a5980f2fbff43b6fbcc051eb597b4d37570e531 2024-09-17T12:15:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 systemd[1]: Started Ceph mon.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:15:36.346 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T12:15:36.346 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: monmap e1: 1 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T12:15:36.346 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: fsmap 2024-09-17T12:15:36.346 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: osdmap e1: 0 total, 0 up, 0 in 2024-09-17T12:15:36.346 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:36 smithi016 ceph-mon[23719]: mgrmap e1: no daemons active 2024-09-17T12:15:37.180 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-17T12:15:37.201 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-17T12:15:37.201 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mgr... 2024-09-17T12:15:37.201 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-17T12:15:37.202 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-17T12:15:37.463 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.y 2024-09-17T12:15:37.464 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-17T12:15:37.619 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5b9f4610-74ee-11ef-bceb-c7b262605968.target.wants/ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@.service. 2024-09-17T12:15:37.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:37 smithi016 systemd[1]: Starting Ceph mgr.y for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:15:37.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:37 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2735571603' entity='client.admin' 2024-09-17T12:15:37.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:37 smithi016 podman[23931]: 2024-09-17 12:15:37.852607471 +0000 UTC m=+0.106410900 container create caffea02f7912ca615199d832616180397687a9a0ef702d231d4eea5f7ef2078 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-17T12:15:37.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:37 smithi016 podman[23931]: 2024-09-17 12:15:37.76262694 +0000 UTC m=+0.016430315 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:reef 2024-09-17T12:15:38.164 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T12:15:38.165 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-09-17T12:15:38.173 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T12:15:38.173 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-17T12:15:38.173 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-09-17T12:15:38.173 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-09-17T12:15:38.183 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T12:15:38.183 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-09-17T12:15:38.191 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T12:15:38.191 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-17T12:15:38.191 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-09-17T12:15:38.191 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-17T12:15:38.191 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr to start... 2024-09-17T12:15:38.191 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr... 2024-09-17T12:15:38.226 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:38 smithi016 podman[23931]: 2024-09-17 12:15:38.079997934 +0000 UTC m=+0.333801295 container init caffea02f7912ca615199d832616180397687a9a0ef702d231d4eea5f7ef2078 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS) 2024-09-17T12:15:38.227 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:38 smithi016 podman[23931]: 2024-09-17 12:15:38.082868879 +0000 UTC m=+0.336672248 container start caffea02f7912ca615199d832616180397687a9a0ef702d231d4eea5f7ef2078 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:15:38.227 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:38 smithi016 bash[23931]: caffea02f7912ca615199d832616180397687a9a0ef702d231d4eea5f7ef2078 2024-09-17T12:15:38.227 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:38 smithi016 systemd[1]: Started Ceph mgr.y for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:15:38.515 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:38.224+0000 7f5e7647b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T12:15:38.515 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:38.284+0000 7f5e7647b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T12:15:38.515 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:38.444+0000 7f5e7647b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T12:15:38.515 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:38.512+0000 7f5e7647b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T12:15:38.822 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:38 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/923455795' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T12:15:38.823 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:38.669+0000 7f5e7647b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:15:38.823 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:38.820+0000 7f5e7647b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T12:15:39.254 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-17T12:15:39.254 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-17T12:15:39.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "5b9f4610-74ee-11ef-bceb-c7b262605968", 2024-09-17T12:15:39.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-09-17T12:15:39.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-17T12:15:39.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-17T12:15:39.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-17T12:15:39.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:39.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-17T12:15:39.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-17T12:15:39.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-09-17T12:15:39.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-17T12:15:39.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-17T12:15:39.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-09-17T12:15:39.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-17T12:15:39.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-17T12:15:39.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:39.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-17T12:15:39.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-17T12:15:39.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-17T12:15:39.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-17T12:15:39.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-17T12:15:39.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-17T12:15:39.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-17T12:15:39.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-17T12:15:39.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-17T12:15:39.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:39.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-17T12:15:39.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T12:15:39.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-17T12:15:39.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-17T12:15:39.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:39.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-17T12:15:39.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-17T12:15:39.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-17T12:15:39.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-17T12:15:39.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-17T12:15:39.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-17T12:15:39.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-09-17T12:15:39.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-17T12:15:39.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T12:15:39.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:39.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-17T12:15:39.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T12:15:39.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-09-17T12:15:25.688870+0000", 2024-09-17T12:15:39.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T12:15:39.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:39.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-17T12:15:39.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-17T12:15:39.259 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (1/15)... 2024-09-17T12:15:39.405 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:39.149+0000 7f5e7647b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T12:15:39.405 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:39.212+0000 7f5e7647b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T12:15:39.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:39.403+0000 7f5e7647b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T12:15:40.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:40 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:40.315+0000 7f5e7647b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T12:15:40.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:40 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:40.384+0000 7f5e7647b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T12:15:41.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:41 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:41.144+0000 7f5e7647b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T12:15:41.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:41 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:41.264+0000 7f5e7647b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T12:15:41.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:41 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:41.414+0000 7f5e7647b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:15:41.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:41 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:41.478+0000 7f5e7647b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T12:15:41.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:41 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:41.630+0000 7f5e7647b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T12:15:41.996 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:41 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:41.701+0000 7f5e7647b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T12:15:41.996 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:41 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:41.763+0000 7f5e7647b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T12:15:41.996 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:41 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:41.926+0000 7f5e7647b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T12:15:41.996 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:41 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:41.993+0000 7f5e7647b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T12:15:42.273 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:42 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:42.062+0000 7f5e7647b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T12:15:42.274 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:42 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:42.148+0000 7f5e7647b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T12:15:42.274 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:42 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:42.210+0000 7f5e7647b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T12:15:42.274 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:42 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:42.271+0000 7f5e7647b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T12:15:43.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:43.087+0000 7f5e7647b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T12:15:43.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-mon[23719]: Activating manager daemon y 2024-09-17T12:15:43.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-mon[23719]: mgrmap e2: y(active, starting, since 0.00285821s) 2024-09-17T12:15:43.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-mon[23719]: from='mgr.14100 172.21.15.16:0/3726600552' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:15:43.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-mon[23719]: from='mgr.14100 172.21.15.16:0/3726600552' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:15:43.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-mon[23719]: from='mgr.14100 172.21.15.16:0/3726600552' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:15:43.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-mon[23719]: from='mgr.14100 172.21.15.16:0/3726600552' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:15:43.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-mon[23719]: from='mgr.14100 172.21.15.16:0/3726600552' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:15:43.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-mon[23719]: Manager daemon y is now available 2024-09-17T12:15:43.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-mon[23719]: from='mgr.14100 172.21.15.16:0/3726600552' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:15:43.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-mon[23719]: from='mgr.14100 172.21.15.16:0/3726600552' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:15:43.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-mon[23719]: from='mgr.14100 172.21.15.16:0/3726600552' entity='mgr.y' 2024-09-17T12:15:43.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-mon[23719]: from='mgr.14100 172.21.15.16:0/3726600552' entity='mgr.y' 2024-09-17T12:15:43.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:43 smithi016 ceph-mon[23719]: from='mgr.14100 172.21.15.16:0/3726600552' entity='mgr.y' 2024-09-17T12:15:45.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:45 smithi016 ceph-mon[23719]: mgrmap e3: y(active, since 1.00785s) 2024-09-17T12:15:45.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:45 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1204053934' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T12:15:45.369 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-17T12:15:45.369 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-17T12:15:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "5b9f4610-74ee-11ef-bceb-c7b262605968", 2024-09-17T12:15:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-09-17T12:15:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-17T12:15:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-17T12:15:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-17T12:15:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-17T12:15:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-17T12:15:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-09-17T12:15:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-17T12:15:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-17T12:15:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-09-17T12:15:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-17T12:15:45.371 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-17T12:15:45.371 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-17T12:15:45.371 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T12:15:45.371 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-17T12:15:45.371 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-17T12:15:45.371 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:45.371 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-17T12:15:45.371 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T12:15:45.371 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-17T12:15:45.371 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-17T12:15:45.371 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-17T12:15:45.371 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-17T12:15:45.374 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-17T12:15:45.375 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-17T12:15:45.375 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:45.375 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-17T12:15:45.375 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-17T12:15:45.375 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-17T12:15:45.375 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-17T12:15:45.375 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-17T12:15:45.375 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-17T12:15:45.375 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-17T12:15:45.375 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-17T12:15:45.375 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-17T12:15:45.375 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:45.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-17T12:15:45.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T12:15:45.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-17T12:15:45.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-17T12:15:45.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:45.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-17T12:15:45.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-17T12:15:45.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-17T12:15:45.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-17T12:15:45.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-17T12:15:45.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-17T12:15:45.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-09-17T12:15:45.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-17T12:15:45.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T12:15:45.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:45.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-17T12:15:45.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T12:15:45.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-09-17T12:15:25.688870+0000", 2024-09-17T12:15:45.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T12:15:45.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-17T12:15:45.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-17T12:15:45.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-17T12:15:45.378 INFO:teuthology.orchestra.run.smithi016.stdout:mgr is available 2024-09-17T12:15:46.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:46 smithi016 ceph-mon[23719]: mgrmap e4: y(active, since 2s) 2024-09-17T12:15:46.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:46 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2649759558' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-17T12:15:46.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:46 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2649759558' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-17T12:15:46.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-17T12:15:46.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-09-17T12:15:46.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:15:46.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-17T12:15:46.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-17T12:15:46.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-17T12:15:46.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-17T12:15:46.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-17T12:15:46.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-17T12:15:46.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-17T12:15:46.486 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-17T12:15:46.486 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-09-17T12:15:46.486 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-17T12:15:46.486 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-17T12:15:46.486 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling cephadm module... 2024-09-17T12:15:47.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:47 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2002743877' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-17T12:15:47.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ignoring --setuser ceph since I am not root 2024-09-17T12:15:47.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ignoring --setgroup ceph since I am not root 2024-09-17T12:15:47.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:47.302+0000 7f017013a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T12:15:47.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:47.362+0000 7f017013a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T12:15:47.787 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:47.517+0000 7f017013a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T12:15:47.787 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:47.587+0000 7f017013a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T12:15:47.787 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:47.739+0000 7f017013a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:15:48.040 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:47.890+0000 7f017013a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T12:15:48.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:48 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2002743877' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-17T12:15:48.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:48 smithi016 ceph-mon[23719]: mgrmap e5: y(active, since 4s) 2024-09-17T12:15:48.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:48 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:48.221+0000 7f017013a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T12:15:48.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:48 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:48.284+0000 7f017013a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T12:15:48.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:48 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:48.478+0000 7f017013a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T12:15:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-17T12:15:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-17T12:15:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-17T12:15:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-17T12:15:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-17T12:15:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-17T12:15:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-09-17T12:15:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 5... 2024-09-17T12:15:49.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:49 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3799370948' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-17T12:15:49.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:49 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:49.397+0000 7f017013a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T12:15:49.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:49 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:49.468+0000 7f017013a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T12:15:50.479 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:50.210+0000 7f017013a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T12:15:50.479 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:50.329+0000 7f017013a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T12:15:50.765 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:50.476+0000 7f017013a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:15:50.765 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:50.540+0000 7f017013a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T12:15:50.765 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:50.692+0000 7f017013a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T12:15:50.765 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:50.762+0000 7f017013a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T12:15:51.055 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:50.824+0000 7f017013a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T12:15:51.055 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:50.986+0000 7f017013a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T12:15:51.055 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:51 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:51.053+0000 7f017013a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T12:15:51.331 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:51 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:51.122+0000 7f017013a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T12:15:51.331 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:51 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:51.207+0000 7f017013a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T12:15:51.331 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:51 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:51.268+0000 7f017013a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T12:15:51.332 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:51 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:51.329+0000 7f017013a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T12:15:52.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:52.137+0000 7f017013a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T12:15:52.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: Active manager daemon y restarted 2024-09-17T12:15:52.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: Activating manager daemon y 2024-09-17T12:15:52.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: osdmap e2: 0 total, 0 up, 0 in 2024-09-17T12:15:52.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: mgrmap e6: y(active, starting, since 0.00331297s) 2024-09-17T12:15:52.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:15:52.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:15:52.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:15:52.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:15:52.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:15:52.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: Manager daemon y is now available 2024-09-17T12:15:52.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:15:52.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:15:52.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:15:52.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:15:52.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:15:52.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:52 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:15:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:53 smithi016 ceph-mon[23719]: Found migration_current of "None". Setting to last migration. 2024-09-17T12:15:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:53 smithi016 ceph-mon[23719]: mgrmap e7: y(active, since 1.00564s) 2024-09-17T12:15:53.644 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-17T12:15:53.644 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-17T12:15:53.644 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-17T12:15:53.644 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-17T12:15:53.645 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 5 is available 2024-09-17T12:15:53.645 INFO:teuthology.orchestra.run.smithi016.stdout:Setting orchestrator backend to cephadm... 2024-09-17T12:15:54.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:54 smithi016 ceph-mon[23719]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-17T12:15:54.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:54 smithi016 ceph-mon[23719]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-17T12:15:55.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:55 smithi016 ceph-mon[23719]: mgrmap e8: y(active, since 2s) 2024-09-17T12:15:55.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:55 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:15:55.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:55 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:15:55.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:55 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:15:55.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:55 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:15:55.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-17T12:15:55.906 INFO:teuthology.orchestra.run.smithi016.stdout:Generating ssh key... 2024-09-17T12:15:56.315 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-mon[23719]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:15:56.315 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:15:56.315 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:56.005+0000 7f00dd625640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:12:15:56] ENGINE Error in HTTPServer.serve 2024-09-17T12:15:56.315 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: Traceback (most recent call last): 2024-09-17T12:15:56.315 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:15:56.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: self._connections.run(self.expiration_interval) 2024-09-17T12:15:56.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:15:56.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: self._run(expiration_interval) 2024-09-17T12:15:56.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:15:56.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:15:56.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:15:56.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:15:56.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:15:56.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: s = self.context.wrap_socket( 2024-09-17T12:15:56.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:15:56.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: return self.sslsocket_class._create( 2024-09-17T12:15:56.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:15:56.317 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: self.do_handshake() 2024-09-17T12:15:56.317 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:15:56.317 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: self._sslobj.do_handshake() 2024-09-17T12:15:56.317 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:15:56.317 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:15:56.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: Generating public/private rsa key pair. 2024-09-17T12:15:56.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: Your identification has been saved in /tmp/tmpw6zxdtbt/key 2024-09-17T12:15:56.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: Your public key has been saved in /tmp/tmpw6zxdtbt/key.pub 2024-09-17T12:15:56.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: The key fingerprint is: 2024-09-17T12:15:56.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: SHA256:KSNGPDbE0p9RxVYc/KrjJVG2bX16EzhK8CEQ4zlPFR8 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:15:56.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: The key's randomart image is: 2024-09-17T12:15:56.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: +---[RSA 3072]----+ 2024-09-17T12:15:56.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: | o. =oo.*+E | 2024-09-17T12:15:56.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: | .oo o + + + . | 2024-09-17T12:15:56.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: | .*. * = .oo | 2024-09-17T12:15:56.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: | o oo +.+o.oo. | 2024-09-17T12:15:56.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: | o o S..o.+o.o| 2024-09-17T12:15:56.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: | . . o ..o...o| 2024-09-17T12:15:56.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: | .o. ...| 2024-09-17T12:15:56.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: | oo ..| 2024-09-17T12:15:56.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: | ... | 2024-09-17T12:15:56.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:15:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: +----[SHA256]-----+ 2024-09-17T12:15:57.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:57 smithi016 ceph-mon[23719]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:15:57.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:57 smithi016 ceph-mon[23719]: [17/Sep/2024:12:15:55] ENGINE Bus STARTING 2024-09-17T12:15:57.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:57 smithi016 ceph-mon[23719]: [17/Sep/2024:12:15:56] ENGINE Serving on https://172.21.15.16:7150 2024-09-17T12:15:57.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:57 smithi016 ceph-mon[23719]: [17/Sep/2024:12:15:56] ENGINE Error in HTTPServer.serve 2024-09-17T12:15:57.505 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:15:57.505 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:15:57.505 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:15:57.505 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:15:57.506 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-09-17T12:15:57.506 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:15:57.506 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:15:57.506 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:15:57.506 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:15:57.506 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:15:57.506 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-09-17T12:15:57.506 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:15:57.506 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-09-17T12:15:57.506 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:15:57.506 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-09-17T12:15:57.506 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:15:57.506 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-09-17T12:15:57.507 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:15:57.507 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:57 smithi016 ceph-mon[23719]: [17/Sep/2024:12:15:56] ENGINE Serving on http://172.21.15.16:8765 2024-09-17T12:15:57.507 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:57 smithi016 ceph-mon[23719]: [17/Sep/2024:12:15:56] ENGINE Bus STARTED 2024-09-17T12:15:57.507 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:57 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:15:57.507 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:57 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:15:58.370 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCRuJ6dfxfS2xOXfgFYYsvJRaKEi0FJP6jUjMA88SU2WO40YtjZozX3Y3OCPI7OdDkFHs+nessGtjnqL2iYH+JBCkiJOztQaedRgryU7e9uRMqwVdnV6uagOZ8omDShSaJlZ+W3R38dspdlOawaOgkPo6OxNZuPmEktgG9EwWmeUxO86OgWIK1Eb6wTwJFAF7SF1XCR6yXRjzBqjcAHF3d1uenIQkjNjcYCERVvGM8VDaf+SzEBPOjP7n2wDccA1S86Z7tOxXWOCixayLGEfCdRRE6wHXs+tl8mZTCCB3ICHwj5+bD5PfJozcvETG0elYvduc5m5em4pqRC/g3yXVij7ZOu3aTE4/HcDXyVONapUICkftK8hP72IFxGddGzImwxfp+s3yrG4NtMgBabl+uhhYX257460Jmybn3iLci3Ue0WjeGJTIj4toGHLI+W1LTM6Cuxt6+2lCNHIZzUqcS3YzYjWmGxtDY8OE636lNZzLdo3jipPpnYdXeEvFuMNZ0= ceph-5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:15:58.370 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-17T12:15:58.371 INFO:teuthology.orchestra.run.smithi016.stdout:Adding key to root@localhost authorized_keys... 2024-09-17T12:15:58.379 INFO:teuthology.orchestra.run.smithi016.stdout:Adding host smithi016... 2024-09-17T12:15:58.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:58 smithi016 ceph-mon[23719]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:15:58.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:58 smithi016 ceph-mon[23719]: Generating ssh key... 2024-09-17T12:15:59.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:15:59 smithi016 ceph-mon[23719]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:16:00.263 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:00 smithi016 ceph-mon[23719]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:16:01.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:01 smithi016 ceph-mon[23719]: Deploying cephadm binary to smithi016 2024-09-17T12:16:02.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Added host 'smithi016' with addr '172.21.15.16' 2024-09-17T12:16:02.762 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mon service... 2024-09-17T12:16:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:03 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:16:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:03 smithi016 ceph-mon[23719]: Added host smithi016 2024-09-17T12:16:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:03 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:16:03.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-17T12:16:03.979 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mgr service... 2024-09-17T12:16:04.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:04 smithi016 ceph-mon[23719]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:16:04.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:04 smithi016 ceph-mon[23719]: Saving service mon spec with placement count:5 2024-09-17T12:16:04.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:04 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:16:05.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:05 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:16:05.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:05 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:16:05.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:05 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:16:05.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-17T12:16:07.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:06 smithi016 ceph-mon[23719]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:16:07.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:06 smithi016 ceph-mon[23719]: Saving service mgr spec with placement count:2 2024-09-17T12:16:07.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:06 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2774757912' entity='client.admin' 2024-09-17T12:16:08.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:08 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:16:08.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:08 smithi016 ceph-mon[23719]: from='mgr.14116 172.21.15.16:0/1254583252' entity='mgr.y' 2024-09-17T12:16:08.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:08 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3032868614' entity='client.admin' 2024-09-17T12:16:08.508 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling the dashboard module... 2024-09-17T12:16:09.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:09 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2760430351' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-17T12:16:09.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:09 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ignoring --setuser ceph since I am not root 2024-09-17T12:16:09.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:09 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ignoring --setgroup ceph since I am not root 2024-09-17T12:16:09.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:09 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:09.485+0000 7f3eaab70040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T12:16:09.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:09 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:09.544+0000 7f3eaab70040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T12:16:10.048 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:09 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:09.700+0000 7f3eaab70040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T12:16:10.049 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:09 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:09.769+0000 7f3eaab70040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T12:16:10.049 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:09 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:09.923+0000 7f3eaab70040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:16:10.336 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:10 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:10.070+0000 7f3eaab70040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T12:16:10.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:10 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2760430351' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-17T12:16:10.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:10 smithi016 ceph-mon[23719]: mgrmap e9: y(active, since 17s) 2024-09-17T12:16:10.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:10 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:10.398+0000 7f3eaab70040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T12:16:10.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:10 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:10.462+0000 7f3eaab70040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T12:16:10.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:10 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:10.661+0000 7f3eaab70040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T12:16:11.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:11 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2905630031' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-17T12:16:11.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-17T12:16:11.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-17T12:16:11.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-17T12:16:11.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-17T12:16:11.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-17T12:16:11.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-17T12:16:11.753 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-09-17T12:16:11.753 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 9... 2024-09-17T12:16:11.873 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:11 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:11.601+0000 7f3eaab70040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T12:16:11.873 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:11 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:11.672+0000 7f3eaab70040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T12:16:12.719 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:12 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:12.449+0000 7f3eaab70040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T12:16:12.720 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:12 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:12.569+0000 7f3eaab70040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T12:16:13.009 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:12 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:12.717+0000 7f3eaab70040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:16:13.009 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:12 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:12.781+0000 7f3eaab70040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T12:16:13.009 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:12 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:12.935+0000 7f3eaab70040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T12:16:13.009 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:13.007+0000 7f3eaab70040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T12:16:13.303 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:13.069+0000 7f3eaab70040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T12:16:13.303 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:13.233+0000 7f3eaab70040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T12:16:13.303 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:13.300+0000 7f3eaab70040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T12:16:13.582 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:13.370+0000 7f3eaab70040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T12:16:13.582 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:13.456+0000 7f3eaab70040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T12:16:13.583 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:13.519+0000 7f3eaab70040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T12:16:13.583 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:13.580+0000 7f3eaab70040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T12:16:14.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:14.399+0000 7f3eaab70040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T12:16:14.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-mon[23719]: Active manager daemon y restarted 2024-09-17T12:16:14.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-mon[23719]: Activating manager daemon y 2024-09-17T12:16:14.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-mon[23719]: osdmap e3: 0 total, 0 up, 0 in 2024-09-17T12:16:14.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-mon[23719]: mgrmap e10: y(active, starting, since 0.00331123s) 2024-09-17T12:16:14.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:16:14.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:16:14.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:16:14.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:16:14.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:16:14.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-mon[23719]: Manager daemon y is now available 2024-09-17T12:16:14.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:16:14.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:16:14.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:14 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:16:16.211 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:16.073+0000 7f3e16805640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:12:16:16] ENGINE Error in HTTPServer.serve 2024-09-17T12:16:16.211 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: Traceback (most recent call last): 2024-09-17T12:16:16.211 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:16:16.211 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: self._connections.run(self.expiration_interval) 2024-09-17T12:16:16.211 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:16:16.211 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: self._run(expiration_interval) 2024-09-17T12:16:16.211 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:16:16.211 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:16:16.211 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:16:16.211 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:16:16.212 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:16:16.212 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: s = self.context.wrap_socket( 2024-09-17T12:16:16.212 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:16:16.212 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: return self.sslsocket_class._create( 2024-09-17T12:16:16.212 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:16:16.212 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: self.do_handshake() 2024-09-17T12:16:16.212 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:16:16.212 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: self._sslobj.do_handshake() 2024-09-17T12:16:16.212 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:16:16.212 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:16:16.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-17T12:16:16.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-17T12:16:16.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-17T12:16:16.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-17T12:16:16.213 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 9 is available 2024-09-17T12:16:16.214 INFO:teuthology.orchestra.run.smithi016.stdout:Generating a dashboard self-signed certificate... 2024-09-17T12:16:16.538 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-mon[23719]: mgrmap e11: y(active, since 1.00612s) 2024-09-17T12:16:16.538 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:16.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-mon[23719]: [17/Sep/2024:12:16:15] ENGINE Bus STARTING 2024-09-17T12:16:16.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-mon[23719]: [17/Sep/2024:12:16:16] ENGINE Serving on https://172.21.15.16:7150 2024-09-17T12:16:16.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-mon[23719]: [17/Sep/2024:12:16:16] ENGINE Error in HTTPServer.serve 2024-09-17T12:16:16.539 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:16:16.539 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:16:16.539 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:16:16.539 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:16:16.539 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-09-17T12:16:16.539 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:16:16.539 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:16:16.539 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:16:16.539 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:16:16.539 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:16:16.540 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-09-17T12:16:16.540 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:16:16.540 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-09-17T12:16:16.540 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:16:16.540 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-09-17T12:16:16.540 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:16:16.540 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-09-17T12:16:16.540 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:16:16.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-mon[23719]: [17/Sep/2024:12:16:16] ENGINE Serving on http://172.21.15.16:8765 2024-09-17T12:16:16.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-mon[23719]: [17/Sep/2024:12:16:16] ENGINE Bus STARTED 2024-09-17T12:16:16.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:16 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:18.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:17 smithi016 ceph-mon[23719]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:16:18.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:17 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:18.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:17 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:18.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:17 smithi016 ceph-mon[23719]: mgrmap e12: y(active, since 2s) 2024-09-17T12:16:18.384 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-17T12:16:18.384 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial admin user... 2024-09-17T12:16:19.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$nbOJ7WQ3HZZXFUyHRPSdTeW2B64Gs32I1EF/MbYOJN1vAKk580DFS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726575379, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-17T12:16:19.819 INFO:teuthology.orchestra.run.smithi016.stdout:Fetching dashboard port number... 2024-09-17T12:16:20.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:20 smithi016 ceph-mon[23719]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:16:20.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:20 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:20.950 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 8443 2024-09-17T12:16:20.961 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T12:16:20.961 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-09-17T12:16:20.968 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T12:16:20.968 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-17T12:16:20.968 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-09-17T12:16:20.968 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-17T12:16:20.970 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph Dashboard is now available at: 2024-09-17T12:16:20.970 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:16:20.971 INFO:teuthology.orchestra.run.smithi016.stdout: URL: https://smithi016.front.sepia.ceph.com:8443/ 2024-09-17T12:16:20.971 INFO:teuthology.orchestra.run.smithi016.stdout: User: admin 2024-09-17T12:16:20.971 INFO:teuthology.orchestra.run.smithi016.stdout: Password: m5vqfakydv 2024-09-17T12:16:20.971 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:16:20.971 INFO:teuthology.orchestra.run.smithi016.stdout:Saving cluster configuration to /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config directory 2024-09-17T12:16:20.971 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling autotune for osd_memory_target 2024-09-17T12:16:21.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:21 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2776187094' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-17T12:16:22.556 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:22 smithi016 ceph-mon[23719]: mgrmap e13: y(active, since 6s) 2024-09-17T12:16:23.387 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-17T12:16:23.387 INFO:teuthology.orchestra.run.smithi016.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-17T12:16:23.387 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:16:23.387 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-17T12:16:23.387 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:16:23.387 INFO:teuthology.orchestra.run.smithi016.stdout:Or, if you are only running a single cluster on this host: 2024-09-17T12:16:23.387 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:16:23.387 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-17T12:16:23.388 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:16:23.388 INFO:teuthology.orchestra.run.smithi016.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-17T12:16:23.388 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:16:23.388 INFO:teuthology.orchestra.run.smithi016.stdout: ceph telemetry on 2024-09-17T12:16:23.388 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:16:23.388 INFO:teuthology.orchestra.run.smithi016.stdout:For more information see: 2024-09-17T12:16:23.388 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:16:23.388 INFO:teuthology.orchestra.run.smithi016.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-17T12:16:23.388 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:16:23.388 INFO:teuthology.orchestra.run.smithi016.stdout:Bootstrap complete. 2024-09-17T12:16:23.423 INFO:tasks.cephadm:Fetching config... 2024-09-17T12:16:23.424 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-17T12:16:23.424 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-17T12:16:23.442 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-17T12:16:23.442 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-17T12:16:23.442 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-17T12:16:23.498 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-17T12:16:23.498 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-17T12:16:23.498 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-17T12:16:23.565 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-17T12:16:23.565 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-17T12:16:23.565 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-17T12:16:23.622 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-17T12:16:23.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCRuJ6dfxfS2xOXfgFYYsvJRaKEi0FJP6jUjMA88SU2WO40YtjZozX3Y3OCPI7OdDkFHs+nessGtjnqL2iYH+JBCkiJOztQaedRgryU7e9uRMqwVdnV6uagOZ8omDShSaJlZ+W3R38dspdlOawaOgkPo6OxNZuPmEktgG9EwWmeUxO86OgWIK1Eb6wTwJFAF7SF1XCR6yXRjzBqjcAHF3d1uenIQkjNjcYCERVvGM8VDaf+SzEBPOjP7n2wDccA1S86Z7tOxXWOCixayLGEfCdRRE6wHXs+tl8mZTCCB3ICHwj5+bD5PfJozcvETG0elYvduc5m5em4pqRC/g3yXVij7ZOu3aTE4/HcDXyVONapUICkftK8hP72IFxGddGzImwxfp+s3yrG4NtMgBabl+uhhYX257460Jmybn3iLci3Ue0WjeGJTIj4toGHLI+W1LTM6Cuxt6+2lCNHIZzUqcS3YzYjWmGxtDY8OE636lNZzLdo3jipPpnYdXeEvFuMNZ0= ceph-5b9f4610-74ee-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T12:16:23.699 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCRuJ6dfxfS2xOXfgFYYsvJRaKEi0FJP6jUjMA88SU2WO40YtjZozX3Y3OCPI7OdDkFHs+nessGtjnqL2iYH+JBCkiJOztQaedRgryU7e9uRMqwVdnV6uagOZ8omDShSaJlZ+W3R38dspdlOawaOgkPo6OxNZuPmEktgG9EwWmeUxO86OgWIK1Eb6wTwJFAF7SF1XCR6yXRjzBqjcAHF3d1uenIQkjNjcYCERVvGM8VDaf+SzEBPOjP7n2wDccA1S86Z7tOxXWOCixayLGEfCdRRE6wHXs+tl8mZTCCB3ICHwj5+bD5PfJozcvETG0elYvduc5m5em4pqRC/g3yXVij7ZOu3aTE4/HcDXyVONapUICkftK8hP72IFxGddGzImwxfp+s3yrG4NtMgBabl+uhhYX257460Jmybn3iLci3Ue0WjeGJTIj4toGHLI+W1LTM6Cuxt6+2lCNHIZzUqcS3YzYjWmGxtDY8OE636lNZzLdo3jipPpnYdXeEvFuMNZ0= ceph-5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:16:23.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCRuJ6dfxfS2xOXfgFYYsvJRaKEi0FJP6jUjMA88SU2WO40YtjZozX3Y3OCPI7OdDkFHs+nessGtjnqL2iYH+JBCkiJOztQaedRgryU7e9uRMqwVdnV6uagOZ8omDShSaJlZ+W3R38dspdlOawaOgkPo6OxNZuPmEktgG9EwWmeUxO86OgWIK1Eb6wTwJFAF7SF1XCR6yXRjzBqjcAHF3d1uenIQkjNjcYCERVvGM8VDaf+SzEBPOjP7n2wDccA1S86Z7tOxXWOCixayLGEfCdRRE6wHXs+tl8mZTCCB3ICHwj5+bD5PfJozcvETG0elYvduc5m5em4pqRC/g3yXVij7ZOu3aTE4/HcDXyVONapUICkftK8hP72IFxGddGzImwxfp+s3yrG4NtMgBabl+uhhYX257460Jmybn3iLci3Ue0WjeGJTIj4toGHLI+W1LTM6Cuxt6+2lCNHIZzUqcS3YzYjWmGxtDY8OE636lNZzLdo3jipPpnYdXeEvFuMNZ0= ceph-5b9f4610-74ee-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T12:16:23.833 INFO:teuthology.orchestra.run.smithi196.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCRuJ6dfxfS2xOXfgFYYsvJRaKEi0FJP6jUjMA88SU2WO40YtjZozX3Y3OCPI7OdDkFHs+nessGtjnqL2iYH+JBCkiJOztQaedRgryU7e9uRMqwVdnV6uagOZ8omDShSaJlZ+W3R38dspdlOawaOgkPo6OxNZuPmEktgG9EwWmeUxO86OgWIK1Eb6wTwJFAF7SF1XCR6yXRjzBqjcAHF3d1uenIQkjNjcYCERVvGM8VDaf+SzEBPOjP7n2wDccA1S86Z7tOxXWOCixayLGEfCdRRE6wHXs+tl8mZTCCB3ICHwj5+bD5PfJozcvETG0elYvduc5m5em4pqRC/g3yXVij7ZOu3aTE4/HcDXyVONapUICkftK8hP72IFxGddGzImwxfp+s3yrG4NtMgBabl+uhhYX257460Jmybn3iLci3Ue0WjeGJTIj4toGHLI+W1LTM6Cuxt6+2lCNHIZzUqcS3YzYjWmGxtDY8OE636lNZzLdo3jipPpnYdXeEvFuMNZ0= ceph-5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:16:23.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-17T12:16:23.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:23 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3663043974' entity='client.admin' 2024-09-17T12:16:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:16:27.700 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-17T12:16:27.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-17T12:16:27.899 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:16:27.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:27 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/842769613' entity='client.admin' 2024-09-17T12:16:27.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:27 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:27.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:27 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:27.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:27 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:16:27.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:27 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:27.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:27 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:16:27.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:27 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:30.216 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi196 2024-09-17T12:16:30.217 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-17T12:16:30.217 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.conf 2024-09-17T12:16:30.235 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-17T12:16:30.235 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:16:30.293 INFO:tasks.cephadm:Adding host smithi196 to orchestrator... 2024-09-17T12:16:30.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch host add smithi196 2024-09-17T12:16:30.502 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:16:30.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:30 smithi016 ceph-mon[23719]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:16:30.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:30.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:16:30.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:16:30.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:16:30.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:30 smithi016 ceph-mon[23719]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:16:30.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:30 smithi016 ceph-mon[23719]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:16:30.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:30 smithi016 ceph-mon[23719]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:16:30.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:30 smithi016 ceph-mon[23719]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:16:30.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:30.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:30.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:32.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:32 smithi016 ceph-mon[23719]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:16:33.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:33 smithi016 ceph-mon[23719]: Deploying cephadm binary to smithi196 2024-09-17T12:16:35.421 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi196' with addr '172.21.15.196' 2024-09-17T12:16:35.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:35 smithi016 ceph-mon[23719]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:36.070 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-17T12:16:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:16:36.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:36 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:36.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:36 smithi016 ceph-mon[23719]: Added host smithi196 2024-09-17T12:16:36.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:36 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:16:36.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:36 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:36.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:36 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:37.906 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:16:37.906 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.196", "hostname": "smithi196", "labels": [], "status": ""}] 2024-09-17T12:16:37.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:37 smithi016 ceph-mon[23719]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:37.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:37 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:38.490 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-17T12:16:38.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-17T12:16:38.686 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:16:38.808 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:38 smithi016 ceph-mon[23719]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T12:16:39.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:39 smithi016 ceph-mon[23719]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:39.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:39 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:39.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:39 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:39.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:39 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:39.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:39 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:39.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:39 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:16:39.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:39 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:16:39.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:39 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:16:41.282 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-09-17T12:16:41.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:41 smithi016 ceph-mon[23719]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:16:41.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:41 smithi016 ceph-mon[23719]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:16:41.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:41 smithi016 ceph-mon[23719]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:16:41.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:41 smithi016 ceph-mon[23719]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:16:41.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:41 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:41.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:41 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:41.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:41 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:41.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:41 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2038100863' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-17T12:16:41.876 INFO:tasks.cephadm:Adding mon.a on smithi016 2024-09-17T12:16:41.876 INFO:tasks.cephadm:Adding mon.c on smithi016 2024-09-17T12:16:41.876 INFO:tasks.cephadm:Adding mon.b on smithi196 2024-09-17T12:16:41.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi016:172.21.15.16=a;smithi016:[v2:172.21.15.16:3301,v1:172.21.15.16:6790]=c;smithi196:172.21.15.196=b' 2024-09-17T12:16:42.073 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:16:42.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:42 smithi016 ceph-mon[23719]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:42.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:42 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2038100863' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-17T12:16:42.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:42 smithi016 ceph-mon[23719]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T12:16:43.840 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mon update... 2024-09-17T12:16:44.619 DEBUG:teuthology.orchestra.run.smithi016:mon.c> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.c.service 2024-09-17T12:16:44.622 DEBUG:teuthology.orchestra.run.smithi196:mon.b> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.b.service 2024-09-17T12:16:44.625 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T12:16:44.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T12:16:44.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:44 smithi016 ceph-mon[23719]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:44.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:44 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:44.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:44 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:16:44.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:44 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:16:44.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:44 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:16:44.649 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:44 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:44.649 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:44 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:16:44.649 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:44 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:16:45.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:45 smithi016 ceph-mon[23719]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi016:[v2:172.21.15.16:3301,v1:172.21.15.16:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:16:45.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:45 smithi016 ceph-mon[23719]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi016:[v2:172.21.15.16:3301,v1:172.21.15.16:6790]=c;smithi196:172.21.15.196=b;count:3 2024-09-17T12:16:45.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:45 smithi016 ceph-mon[23719]: Deploying daemon mon.b on smithi196 2024-09-17T12:16:46.177 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:16:46.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:46 smithi016 ceph-mon[23719]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:48.656 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-17T12:16:48.656 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"5b9f4610-74ee-11ef-bceb-c7b262605968","modified":"2024-09-17T12:15:22.758286Z","created":"2024-09-17T12:15:22.758286Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-17T12:16:48.656 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-09-17T12:16:48.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:48 smithi016 ceph-mon[23719]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:48.820 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 podman[27396]: 2024-09-17 12:16:48.796427315 +0000 UTC m=+0.355834952 container start 262ef622777e9a56bb7eda009090d057f7f541d41b5dd406cfc1f48fe8ee21f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-17T12:16:48.821 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T12:16:48.821 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process ceph-mon, pid 2 2024-09-17T12:16:48.821 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: pidfile_write: ignore empty --pid-file 2024-09-17T12:16:49.255 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: load: jerasure load: lrc 2024-09-17T12:16:49.256 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: RocksDB version: 7.9.2 2024-09-17T12:16:49.256 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Git sha 0 2024-09-17T12:16:49.256 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Compile date 2024-09-16 15:15:13 2024-09-17T12:16:49.256 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: DB SUMMARY 2024-09-17T12:16:49.256 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: DB Session ID: GILPX6QUESU0Z2UGSHP4 2024-09-17T12:16:49.256 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: CURRENT file: CURRENT 2024-09-17T12:16:49.257 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T12:16:49.257 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-17T12:16:49.257 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-17T12:16:49.257 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-17T12:16:49.257 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.error_if_exists: 0 2024-09-17T12:16:49.257 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.create_if_missing: 0 2024-09-17T12:16:49.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.paranoid_checks: 1 2024-09-17T12:16:49.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T12:16:49.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T12:16:49.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T12:16:49.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.env: 0x5579352f2bc0 2024-09-17T12:16:49.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T12:16:49.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.info_log: 0x557937b16be0 2024-09-17T12:16:49.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T12:16:49.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.statistics: (nil) 2024-09-17T12:16:49.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.use_fsync: 0 2024-09-17T12:16:49.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_log_file_size: 0 2024-09-17T12:16:49.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T12:16:49.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T12:16:49.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T12:16:49.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T12:16:49.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.allow_fallocate: 1 2024-09-17T12:16:49.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T12:16:49.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T12:16:49.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.use_direct_reads: 0 2024-09-17T12:16:49.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T12:16:49.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T12:16:49.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.db_log_dir: 2024-09-17T12:16:49.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.wal_dir: 2024-09-17T12:16:49.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T12:16:49.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T12:16:49.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T12:16:49.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T12:16:49.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T12:16:49.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T12:16:49.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T12:16:49.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T12:16:49.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.write_buffer_manager: 0x557937b24320 2024-09-17T12:16:49.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T12:16:49.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T12:16:49.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T12:16:49.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T12:16:49.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T12:16:49.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T12:16:49.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T12:16:49.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T12:16:49.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.unordered_write: 0 2024-09-17T12:16:49.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T12:16:49.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T12:16:49.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T12:16:49.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T12:16:49.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.row_cache: None 2024-09-17T12:16:49.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.wal_filter: None 2024-09-17T12:16:49.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T12:16:49.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T12:16:49.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.two_write_queues: 0 2024-09-17T12:16:49.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T12:16:49.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.wal_compression: 0 2024-09-17T12:16:49.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.atomic_flush: 0 2024-09-17T12:16:49.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T12:16:49.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T12:16:49.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T12:16:49.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.log_readahead_size: 0 2024-09-17T12:16:49.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T12:16:49.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T12:16:49.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T12:16:49.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T12:16:49.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T12:16:49.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T12:16:49.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T12:16:49.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_background_jobs: 2 2024-09-17T12:16:49.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_background_compactions: -1 2024-09-17T12:16:49.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_subcompactions: 1 2024-09-17T12:16:49.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T12:16:49.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T12:16:49.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T12:16:49.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T12:16:49.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T12:16:49.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T12:16:49.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T12:16:49.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T12:16:49.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_open_files: -1 2024-09-17T12:16:49.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T12:16:49.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T12:16:49.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T12:16:49.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T12:16:49.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_background_flushes: -1 2024-09-17T12:16:49.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Compression algorithms supported: 2024-09-17T12:16:49.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: kZSTD supported: 0 2024-09-17T12:16:49.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: kXpressCompression supported: 0 2024-09-17T12:16:49.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: kBZip2Compression supported: 0 2024-09-17T12:16:49.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T12:16:49.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: kLZ4Compression supported: 1 2024-09-17T12:16:49.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: kZlibCompression supported: 1 2024-09-17T12:16:49.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T12:16:49.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: kSnappyCompression supported: 1 2024-09-17T12:16:49.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T12:16:49.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T12:16:49.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-17T12:16:49.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T12:16:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T12:16:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.merge_operator: 2024-09-17T12:16:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compaction_filter: None 2024-09-17T12:16:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T12:16:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T12:16:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T12:16:49.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T12:16:49.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557937b167e0) 2024-09-17T12:16:49.278 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks: 1 2024-09-17T12:16:49.278 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T12:16:49.278 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T12:16:49.278 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_top_level_index_and_filter: 1 2024-09-17T12:16:49.279 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_type: 0 2024-09-17T12:16:49.279 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_index_type: 0 2024-09-17T12:16:49.279 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_shortening: 1 2024-09-17T12:16:49.279 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T12:16:49.279 INFO:journalctl@ceph.mon.b.smithi196.stdout: checksum: 4 2024-09-17T12:16:49.280 INFO:journalctl@ceph.mon.b.smithi196.stdout: no_block_cache: 0 2024-09-17T12:16:49.280 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache: 0x557937b0b350 2024-09-17T12:16:49.280 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_name: BinnedLRUCache 2024-09-17T12:16:49.280 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_options: 2024-09-17T12:16:49.280 INFO:journalctl@ceph.mon.b.smithi196.stdout: capacity : 536870912 2024-09-17T12:16:49.280 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_shard_bits : 4 2024-09-17T12:16:49.281 INFO:journalctl@ceph.mon.b.smithi196.stdout: strict_capacity_limit : 0 2024-09-17T12:16:49.281 INFO:journalctl@ceph.mon.b.smithi196.stdout: high_pri_pool_ratio: 0.000 2024-09-17T12:16:49.281 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_compressed: (nil) 2024-09-17T12:16:49.281 INFO:journalctl@ceph.mon.b.smithi196.stdout: persistent_cache: (nil) 2024-09-17T12:16:49.281 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size: 4096 2024-09-17T12:16:49.282 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size_deviation: 10 2024-09-17T12:16:49.282 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_restart_interval: 16 2024-09-17T12:16:49.282 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_block_restart_interval: 1 2024-09-17T12:16:49.282 INFO:journalctl@ceph.mon.b.smithi196.stdout: metadata_block_size: 4096 2024-09-17T12:16:49.282 INFO:journalctl@ceph.mon.b.smithi196.stdout: partition_filters: 0 2024-09-17T12:16:49.282 INFO:journalctl@ceph.mon.b.smithi196.stdout: use_delta_encoding: 1 2024-09-17T12:16:49.283 INFO:journalctl@ceph.mon.b.smithi196.stdout: filter_policy: bloomfilter 2024-09-17T12:16:49.283 INFO:journalctl@ceph.mon.b.smithi196.stdout: whole_key_filtering: 1 2024-09-17T12:16:49.283 INFO:journalctl@ceph.mon.b.smithi196.stdout: verify_compression: 0 2024-09-17T12:16:49.283 INFO:journalctl@ceph.mon.b.smithi196.stdout: read_amp_bytes_per_bit: 0 2024-09-17T12:16:49.283 INFO:journalctl@ceph.mon.b.smithi196.stdout: format_version: 5 2024-09-17T12:16:49.283 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_index_compression: 1 2024-09-17T12:16:49.283 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_align: 0 2024-09-17T12:16:49.284 INFO:journalctl@ceph.mon.b.smithi196.stdout: max_auto_readahead_size: 262144 2024-09-17T12:16:49.284 INFO:journalctl@ceph.mon.b.smithi196.stdout: prepopulate_block_cache: 0 2024-09-17T12:16:49.284 INFO:journalctl@ceph.mon.b.smithi196.stdout: initial_auto_readahead_size: 8192 2024-09-17T12:16:49.284 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T12:16:49.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T12:16:49.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T12:16:49.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compression: NoCompression 2024-09-17T12:16:49.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T12:16:49.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T12:16:49.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T12:16:49.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.num_levels: 7 2024-09-17T12:16:49.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T12:16:49.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T12:16:49.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T12:16:49.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T12:16:49.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T12:16:49.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T12:16:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T12:16:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T12:16:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T12:16:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T12:16:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T12:16:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T12:16:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T12:16:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T12:16:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T12:16:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T12:16:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T12:16:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T12:16:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T12:16:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T12:16:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T12:16:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T12:16:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T12:16:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T12:16:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T12:16:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T12:16:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T12:16:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T12:16:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T12:16:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T12:16:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T12:16:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T12:16:49.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T12:16:49.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T12:16:49.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T12:16:49.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T12:16:49.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T12:16:49.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T12:16:49.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T12:16:49.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T12:16:49.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T12:16:49.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T12:16:49.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T12:16:49.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T12:16:49.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T12:16:49.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T12:16:49.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T12:16:49.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T12:16:49.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T12:16:49.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T12:16:49.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T12:16:49.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T12:16:49.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T12:16:49.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T12:16:49.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.inplace_update_support: 0 2024-09-17T12:16:49.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T12:16:49.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T12:16:49.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T12:16:49.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T12:16:49.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.bloom_locality: 0 2024-09-17T12:16:49.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.max_successive_merges: 0 2024-09-17T12:16:49.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T12:16:49.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T12:16:49.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T12:16:49.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T12:16:49.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.ttl: 2592000 2024-09-17T12:16:49.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T12:16:49.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T12:16:49.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T12:16:49.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.enable_blob_files: false 2024-09-17T12:16:49.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.min_blob_size: 0 2024-09-17T12:16:49.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T12:16:49.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T12:16:49.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T12:16:49.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T12:16:49.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T12:16:49.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T12:16:49.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T12:16:49.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T12:16:49.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T12:16:49.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-17T12:16:49.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 566be203-b006-41f1-ba60-2c82ca3e7321 2024-09-17T12:16:49.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726575408824802, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-17T12:16:49.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-17T12:16:49.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726575408825198, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726575408, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "566be203-b006-41f1-ba60-2c82ca3e7321", "db_session_id": "GILPX6QUESU0Z2UGSHP4", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-17T12:16:49.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726575408825293, "job": 1, "event": "recovery_finished"} 2024-09-17T12:16:49.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-17T12:16:49.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T12:16:49.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557937b36e00 2024-09-17T12:16:49.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: DB pointer 0x557937c2a000 2024-09-17T12:16:49.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T12:16:49.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T12:16:49.293 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** DB Stats ** 2024-09-17T12:16:49.293 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T12:16:49.294 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T12:16:49.294 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T12:16:49.294 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T12:16:49.294 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T12:16:49.294 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T12:16:49.294 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T12:16:49.294 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-17T12:16:49.294 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-17T12:16:49.294 INFO:journalctl@ceph.mon.b.smithi196.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T12:16:49.294 INFO:journalctl@ceph.mon.b.smithi196.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T12:16:49.294 INFO:journalctl@ceph.mon.b.smithi196.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T12:16:49.294 INFO:journalctl@ceph.mon.b.smithi196.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi196.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-17T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-17T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi196.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi196.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi196.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-17T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi196.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-17T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi196.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T12:16:49.296 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T12:16:49.296 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T12:16:49.296 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T12:16:49.296 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T12:16:49.296 INFO:journalctl@ceph.mon.b.smithi196.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T12:16:49.296 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache BinnedLRUCache@0x557937b0b350#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-09-17T12:16:49.296 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-17T12:16:49.296 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-17T12:16:49.296 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T12:16:49.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-17T12:16:49.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: using public_addr v2:172.21.15.196:0/0 -> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] 2024-09-17T12:16:49.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: starting mon.b rank -1 at public addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] at bind addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:16:49.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(???) e0 preinit fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:16:49.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(synchronizing).mds e1 new map 2024-09-17T12:16:49.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-17T12:16:49.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: e1 2024-09-17T12:16:49.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T12:16:49.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T12:16:49.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: legacy client fscid: -1 2024-09-17T12:16:49.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-17T12:16:49.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: No filesystems configured 2024-09-17T12:16:49.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi016:[v2:172.21.15.16:3301,v1:172.21.15.16:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi016:[v2:172.21.15.16:3301,v1:172.21.15.16:6790]=c;smithi196:172.21.15.196=b;count:3 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: Deploying daemon mon.b on smithi196 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:49.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:49.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 ceph-mon[27428]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-17T12:16:49.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 bash[27396]: 262ef622777e9a56bb7eda009090d057f7f541d41b5dd406cfc1f48fe8ee21f8 2024-09-17T12:16:49.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:48 smithi196 systemd[1]: Started Ceph mon.b for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:16:50.615 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T12:16:50.616 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T12:16:50.834 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:16:54.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:54.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:16:54.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: mon.a calling monitor election 2024-09-17T12:16:54.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:54.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:54.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:54.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:54.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: mon.b calling monitor election 2024-09-17T12:16:54.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:54.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:54.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:54.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:54.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-17T12:16:54.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: monmap e2: 2 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T12:16:54.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: fsmap 2024-09-17T12:16:54.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T12:16:54.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: mgrmap e13: y(active, since 39s) 2024-09-17T12:16:54.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: overall HEALTH_OK 2024-09-17T12:16:54.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:54.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:54.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:54.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:16:54.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:53 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:16:54.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:54.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:16:54.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: mon.a calling monitor election 2024-09-17T12:16:54.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:54.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:54.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:54.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:54.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: mon.b calling monitor election 2024-09-17T12:16:54.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:54.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:54.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:54.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:54.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-17T12:16:54.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: monmap e2: 2 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T12:16:54.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: fsmap 2024-09-17T12:16:54.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T12:16:54.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: mgrmap e13: y(active, since 39s) 2024-09-17T12:16:54.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: overall HEALTH_OK 2024-09-17T12:16:54.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:54.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:54.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:54.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:16:54.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:53 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:16:55.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:54 smithi016 ceph-mon[23719]: Deploying daemon mon.c on smithi016 2024-09-17T12:16:55.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:54 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:55.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:54 smithi196 ceph-mon[27428]: Deploying daemon mon.c on smithi016 2024-09-17T12:16:55.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:54 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-17T12:16:55.409 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":2,"fsid":"5b9f4610-74ee-11ef-bceb-c7b262605968","modified":"2024-09-17T12:16:48.852775Z","created":"2024-09-17T12:15:22.758286Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T12:16:55.409 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 2 2024-09-17T12:16:56.010 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:55 smithi196 ceph-mon[27428]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:56.010 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:16:55 smithi196 ceph-mon[27428]: from='client.? 172.21.15.196:0/2298286038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T12:16:56.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:55 smithi016 ceph-mon[23719]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:56.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:16:55 smithi016 ceph-mon[23719]: from='client.? 172.21.15.196:0/2298286038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T12:16:56.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:56 smithi016 systemd[1]: Starting Ceph mon.c for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:16:57.031 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T12:16:57.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T12:16:57.209 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:56 smithi016 podman[34192]: 2024-09-17 12:16:56.859371308 +0000 UTC m=+0.016313406 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T12:16:57.209 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:56 smithi016 podman[34192]: 2024-09-17 12:16:56.96145932 +0000 UTC m=+0.118401407 container create 950b726343643eb0826912ae44363bc341a79460cadbcd27937cd85d9a55afe5 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux ) 2024-09-17T12:16:57.209 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 podman[34192]: 2024-09-17 12:16:57.173235317 +0000 UTC m=+0.330177413 container init 950b726343643eb0826912ae44363bc341a79460cadbcd27937cd85d9a55afe5 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True) 2024-09-17T12:16:57.209 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 podman[34192]: 2024-09-17 12:16:57.177727699 +0000 UTC m=+0.334669790 container start 950b726343643eb0826912ae44363bc341a79460cadbcd27937cd85d9a55afe5 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-17T12:16:57.209 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T12:16:57.209 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process ceph-mon, pid 2 2024-09-17T12:16:57.209 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: pidfile_write: ignore empty --pid-file 2024-09-17T12:16:57.209 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: load: jerasure load: lrc 2024-09-17T12:16:57.209 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: RocksDB version: 7.9.2 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Git sha 0 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Compile date 2024-09-16 15:15:13 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: DB SUMMARY 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: DB Session ID: FXHUFRGSYEVJJGRNDDN7 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: CURRENT file: CURRENT 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.error_if_exists: 0 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.create_if_missing: 0 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.paranoid_checks: 1 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T12:16:57.210 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T12:16:57.211 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T12:16:57.211 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.env: 0x55627d4b6bc0 2024-09-17T12:16:57.211 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T12:16:57.211 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.info_log: 0x55627f230be0 2024-09-17T12:16:57.211 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T12:16:57.211 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.statistics: (nil) 2024-09-17T12:16:57.211 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.use_fsync: 0 2024-09-17T12:16:57.211 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_log_file_size: 0 2024-09-17T12:16:57.211 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T12:16:57.212 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T12:16:57.212 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T12:16:57.212 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T12:16:57.212 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.allow_fallocate: 1 2024-09-17T12:16:57.212 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T12:16:57.212 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T12:16:57.212 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.use_direct_reads: 0 2024-09-17T12:16:57.212 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T12:16:57.212 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T12:16:57.212 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.db_log_dir: 2024-09-17T12:16:57.212 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.wal_dir: 2024-09-17T12:16:57.212 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T12:16:57.213 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T12:16:57.213 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T12:16:57.213 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T12:16:57.213 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T12:16:57.213 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T12:16:57.213 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T12:16:57.213 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T12:16:57.213 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.write_buffer_manager: 0x55627f23e320 2024-09-17T12:16:57.213 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T12:16:57.213 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T12:16:57.213 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T12:16:57.213 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T12:16:57.213 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T12:16:57.214 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T12:16:57.214 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T12:16:57.214 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T12:16:57.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.unordered_write: 0 2024-09-17T12:16:57.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T12:16:57.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T12:16:57.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T12:16:57.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T12:16:57.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.row_cache: None 2024-09-17T12:16:57.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.wal_filter: None 2024-09-17T12:16:57.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T12:16:57.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T12:16:57.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.two_write_queues: 0 2024-09-17T12:16:57.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T12:16:57.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.wal_compression: 0 2024-09-17T12:16:57.216 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.atomic_flush: 0 2024-09-17T12:16:57.216 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T12:16:57.216 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T12:16:57.217 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T12:16:57.217 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.log_readahead_size: 0 2024-09-17T12:16:57.217 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T12:16:57.217 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T12:16:57.217 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T12:16:57.218 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T12:16:57.218 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T12:16:57.218 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T12:16:57.218 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T12:16:57.218 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_background_jobs: 2 2024-09-17T12:16:57.219 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_background_compactions: -1 2024-09-17T12:16:57.219 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_subcompactions: 1 2024-09-17T12:16:57.219 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T12:16:57.219 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T12:16:57.219 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T12:16:57.219 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T12:16:57.220 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T12:16:57.220 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T12:16:57.220 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T12:16:57.220 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T12:16:57.220 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_open_files: -1 2024-09-17T12:16:57.220 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T12:16:57.220 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T12:16:57.221 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T12:16:57.221 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T12:16:57.221 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_background_flushes: -1 2024-09-17T12:16:57.221 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Compression algorithms supported: 2024-09-17T12:16:57.222 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: kZSTD supported: 0 2024-09-17T12:16:57.222 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: kXpressCompression supported: 0 2024-09-17T12:16:57.222 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: kBZip2Compression supported: 0 2024-09-17T12:16:57.222 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T12:16:57.222 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: kLZ4Compression supported: 1 2024-09-17T12:16:57.222 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: kZlibCompression supported: 1 2024-09-17T12:16:57.223 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T12:16:57.223 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: kSnappyCompression supported: 1 2024-09-17T12:16:57.223 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T12:16:57.223 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T12:16:57.223 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-17T12:16:57.224 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T12:16:57.224 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T12:16:57.224 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.merge_operator: 2024-09-17T12:16:57.224 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compaction_filter: None 2024-09-17T12:16:57.224 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T12:16:57.225 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T12:16:57.251 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:16:57.514 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T12:16:57.514 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T12:16:57.514 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55627f2307e0) 2024-09-17T12:16:57.514 INFO:journalctl@ceph.mon.c.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-09-17T12:16:57.514 INFO:journalctl@ceph.mon.c.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T12:16:57.514 INFO:journalctl@ceph.mon.c.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T12:16:57.515 INFO:journalctl@ceph.mon.c.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-09-17T12:16:57.515 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_type: 0 2024-09-17T12:16:57.515 INFO:journalctl@ceph.mon.c.smithi016.stdout: data_block_index_type: 0 2024-09-17T12:16:57.515 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_shortening: 1 2024-09-17T12:16:57.515 INFO:journalctl@ceph.mon.c.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T12:16:57.515 INFO:journalctl@ceph.mon.c.smithi016.stdout: checksum: 4 2024-09-17T12:16:57.515 INFO:journalctl@ceph.mon.c.smithi016.stdout: no_block_cache: 0 2024-09-17T12:16:57.515 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache: 0x55627f2251f0 2024-09-17T12:16:57.515 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-09-17T12:16:57.515 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_options: 2024-09-17T12:16:57.516 INFO:journalctl@ceph.mon.c.smithi016.stdout: capacity : 536870912 2024-09-17T12:16:57.516 INFO:journalctl@ceph.mon.c.smithi016.stdout: num_shard_bits : 4 2024-09-17T12:16:57.516 INFO:journalctl@ceph.mon.c.smithi016.stdout: strict_capacity_limit : 0 2024-09-17T12:16:57.516 INFO:journalctl@ceph.mon.c.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-09-17T12:16:57.516 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_compressed: (nil) 2024-09-17T12:16:57.516 INFO:journalctl@ceph.mon.c.smithi016.stdout: persistent_cache: (nil) 2024-09-17T12:16:57.516 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_size: 4096 2024-09-17T12:16:57.516 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_size_deviation: 10 2024-09-17T12:16:57.516 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_restart_interval: 16 2024-09-17T12:16:57.516 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_block_restart_interval: 1 2024-09-17T12:16:57.516 INFO:journalctl@ceph.mon.c.smithi016.stdout: metadata_block_size: 4096 2024-09-17T12:16:57.517 INFO:journalctl@ceph.mon.c.smithi016.stdout: partition_filters: 0 2024-09-17T12:16:57.517 INFO:journalctl@ceph.mon.c.smithi016.stdout: use_delta_encoding: 1 2024-09-17T12:16:57.517 INFO:journalctl@ceph.mon.c.smithi016.stdout: filter_policy: bloomfilter 2024-09-17T12:16:57.517 INFO:journalctl@ceph.mon.c.smithi016.stdout: whole_key_filtering: 1 2024-09-17T12:16:57.517 INFO:journalctl@ceph.mon.c.smithi016.stdout: verify_compression: 0 2024-09-17T12:16:57.517 INFO:journalctl@ceph.mon.c.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-09-17T12:16:57.517 INFO:journalctl@ceph.mon.c.smithi016.stdout: format_version: 5 2024-09-17T12:16:57.517 INFO:journalctl@ceph.mon.c.smithi016.stdout: enable_index_compression: 1 2024-09-17T12:16:57.517 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_align: 0 2024-09-17T12:16:57.517 INFO:journalctl@ceph.mon.c.smithi016.stdout: max_auto_readahead_size: 262144 2024-09-17T12:16:57.517 INFO:journalctl@ceph.mon.c.smithi016.stdout: prepopulate_block_cache: 0 2024-09-17T12:16:57.517 INFO:journalctl@ceph.mon.c.smithi016.stdout: initial_auto_readahead_size: 8192 2024-09-17T12:16:57.518 INFO:journalctl@ceph.mon.c.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T12:16:57.518 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T12:16:57.518 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T12:16:57.518 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compression: NoCompression 2024-09-17T12:16:57.518 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T12:16:57.518 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T12:16:57.518 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T12:16:57.518 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.num_levels: 7 2024-09-17T12:16:57.518 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T12:16:57.518 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T12:16:57.518 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T12:16:57.519 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T12:16:57.519 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T12:16:57.519 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T12:16:57.519 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T12:16:57.519 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T12:16:57.519 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T12:16:57.519 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T12:16:57.519 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T12:16:57.519 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T12:16:57.520 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T12:16:57.520 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T12:16:57.520 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T12:16:57.520 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T12:16:57.520 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T12:16:57.520 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T12:16:57.520 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T12:16:57.520 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T12:16:57.520 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T12:16:57.520 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T12:16:57.520 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T12:16:57.521 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T12:16:57.521 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T12:16:57.521 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T12:16:57.521 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T12:16:57.521 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T12:16:57.521 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T12:16:57.521 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T12:16:57.522 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T12:16:57.522 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T12:16:57.522 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T12:16:57.522 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T12:16:57.522 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T12:16:57.522 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T12:16:57.523 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T12:16:57.523 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T12:16:57.523 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T12:16:57.523 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T12:16:57.523 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T12:16:57.524 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T12:16:57.524 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T12:16:57.524 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T12:16:57.524 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T12:16:57.524 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T12:16:57.524 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T12:16:57.524 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T12:16:57.524 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T12:16:57.524 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T12:16:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T12:16:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T12:16:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T12:16:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T12:16:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.inplace_update_support: 0 2024-09-17T12:16:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T12:16:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T12:16:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T12:16:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T12:16:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.bloom_locality: 0 2024-09-17T12:16:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.max_successive_merges: 0 2024-09-17T12:16:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.ttl: 2592000 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.enable_blob_files: false 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.min_blob_size: 0 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T12:16:57.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T12:16:57.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T12:16:57.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T12:16:57.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T12:16:57.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T12:16:57.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-17T12:16:57.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: beb14d0f-605c-4ec2-818d-c14fa1c35b70 2024-09-17T12:16:57.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726575417205882, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-17T12:16:57.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-17T12:16:57.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726575417206294, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726575417, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "beb14d0f-605c-4ec2-818d-c14fa1c35b70", "db_session_id": "FXHUFRGSYEVJJGRNDDN7", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-17T12:16:57.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726575417206392, "job": 1, "event": "recovery_finished"} 2024-09-17T12:16:57.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-17T12:16:57.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T12:16:57.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55627f250e00 2024-09-17T12:16:57.528 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: DB pointer 0x55627f344000 2024-09-17T12:16:57.528 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T12:16:57.528 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T12:16:57.528 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** DB Stats ** 2024-09-17T12:16:57.528 INFO:journalctl@ceph.mon.c.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T12:16:57.528 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T12:16:57.528 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T12:16:57.528 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T12:16:57.528 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T12:16:57.528 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T12:16:57.528 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T12:16:57.528 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-09-17T12:16:57.528 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** Compaction Stats [default] ** 2024-09-17T12:16:57.529 INFO:journalctl@ceph.mon.c.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T12:16:57.529 INFO:journalctl@ceph.mon.c.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T12:16:57.529 INFO:journalctl@ceph.mon.c.smithi016.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T12:16:57.529 INFO:journalctl@ceph.mon.c.smithi016.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T12:16:57.529 INFO:journalctl@ceph.mon.c.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T12:16:57.529 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-09-17T12:16:57.529 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** Compaction Stats [default] ** 2024-09-17T12:16:57.529 INFO:journalctl@ceph.mon.c.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T12:16:57.529 INFO:journalctl@ceph.mon.c.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T12:16:57.529 INFO:journalctl@ceph.mon.c.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T12:16:57.529 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-09-17T12:16:57.529 INFO:journalctl@ceph.mon.c.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: Block cache BinnedLRUCache@0x55627f2251f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-09-17T12:16:57.530 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T12:16:57.531 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-17T12:16:57.531 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: using public_addrv [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] 2024-09-17T12:16:57.531 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: starting mon.c rank -1 at public addrs [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] at bind addrs [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:16:57.531 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(???) e0 preinit fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:16:57.531 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(synchronizing).mds e1 new map 2024-09-17T12:16:57.531 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-17T12:16:57.531 INFO:journalctl@ceph.mon.c.smithi016.stdout: e1 2024-09-17T12:16:57.531 INFO:journalctl@ceph.mon.c.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T12:16:57.531 INFO:journalctl@ceph.mon.c.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T12:16:57.531 INFO:journalctl@ceph.mon.c.smithi016.stdout: legacy client fscid: -1 2024-09-17T12:16:57.531 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-09-17T12:16:57.531 INFO:journalctl@ceph.mon.c.smithi016.stdout: No filesystems configured 2024-09-17T12:16:57.532 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-17T12:16:57.532 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-17T12:16:57.532 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-17T12:16:57.532 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-17T12:16:57.532 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-17T12:16:57.532 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-17T12:16:57.532 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T12:16:57.532 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T12:16:57.532 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T12:16:57.532 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T12:16:57.532 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi016:[v2:172.21.15.16:3301,v1:172.21.15.16:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:16:57.532 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi016:[v2:172.21.15.16:3301,v1:172.21.15.16:6790]=c;smithi196:172.21.15.196=b;count:3 2024-09-17T12:16:57.532 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: Deploying daemon mon.b on smithi196 2024-09-17T12:16:57.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:57.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:57.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:57.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:16:57.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.a calling monitor election 2024-09-17T12:16:57.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:57.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:57.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:57.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:57.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.b calling monitor election 2024-09-17T12:16:57.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:57.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:57.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:57.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:57.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-17T12:16:57.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: monmap e2: 2 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T12:16:57.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: fsmap 2024-09-17T12:16:57.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T12:16:57.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mgrmap e13: y(active, since 39s) 2024-09-17T12:16:57.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: overall HEALTH_OK 2024-09-17T12:16:57.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:57.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:57.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:16:57.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:16:57.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:16:57.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: Deploying daemon mon.c on smithi016 2024-09-17T12:16:57.535 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:16:57.535 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:16:57.535 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: from='client.? 172.21.15.196:0/2298286038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T12:16:57.535 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 ceph-mon[34212]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-17T12:16:57.535 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 bash[34192]: 950b726343643eb0826912ae44363bc341a79460cadbcd27937cd85d9a55afe5 2024-09-17T12:16:57.535 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:16:57 smithi016 systemd[1]: Started Ceph mon.c for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:17:02.530 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-17T12:17:02.530 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":3,"fsid":"5b9f4610-74ee-11ef-bceb-c7b262605968","modified":"2024-09-17T12:16:57.230811Z","created":"2024-09-17T12:15:22.758286Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3301","nonce":0},{"type":"v1","addr":"172.21.15.16:6790","nonce":0}]},"addr":"172.21.15.16:6790/0","public_addr":"172.21.15.16:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-17T12:17:02.530 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 3 2024-09-17T12:17:02.631 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:02.631 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:17:02.631 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: mon.a calling monitor election 2024-09-17T12:17:02.631 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:17:02.631 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: mon.b calling monitor election 2024-09-17T12:17:02.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:02.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: mon.c calling monitor election 2024-09-17T12:17:02.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:02.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T12:17:02.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: monmap e3: 3 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T12:17:02.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: fsmap 2024-09-17T12:17:02.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T12:17:02.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: mgrmap e13: y(active, since 47s) 2024-09-17T12:17:02.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: overall HEALTH_OK 2024-09-17T12:17:02.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:02.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:02.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:02.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:02.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:02.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:17:02.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: mon.a calling monitor election 2024-09-17T12:17:02.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:17:02.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: mon.b calling monitor election 2024-09-17T12:17:02.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:02.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: mon.c calling monitor election 2024-09-17T12:17:02.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:02.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T12:17:02.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: monmap e3: 3 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T12:17:02.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: fsmap 2024-09-17T12:17:02.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T12:17:02.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: mgrmap e13: y(active, since 47s) 2024-09-17T12:17:02.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: overall HEALTH_OK 2024-09-17T12:17:02.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:02.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:02.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:02.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:02.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:02.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:17:02.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: mon.a calling monitor election 2024-09-17T12:17:02.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:17:02.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: mon.b calling monitor election 2024-09-17T12:17:02.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:02.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: mon.c calling monitor election 2024-09-17T12:17:02.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:02.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:02.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T12:17:02.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: monmap e3: 3 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T12:17:02.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: fsmap 2024-09-17T12:17:02.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T12:17:02.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: mgrmap e13: y(active, since 47s) 2024-09-17T12:17:02.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: overall HEALTH_OK 2024-09-17T12:17:02.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:02.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:02.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:02.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:03.143 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-17T12:17:03.143 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-17T12:17:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:17:03.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:03 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:03.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:03 smithi196 ceph-mon[27428]: from='client.? 172.21.15.196:0/2916136067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T12:17:03.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:03 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:03 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:03 smithi016 ceph-mon[23719]: from='client.? 172.21.15.196:0/2916136067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T12:17:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:03 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:03.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:03 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:03.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:03 smithi016 ceph-mon[34212]: from='client.? 172.21.15.196:0/2916136067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T12:17:03.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:03 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:17:04.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[23719]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:04.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:04.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:04.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.554 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[34212]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:04.554 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.554 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.554 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.554 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.554 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:04.554 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:04.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:04 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:04 smithi196 ceph-mon[27428]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:04.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:04 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:04 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:04 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:04 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:04 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:04.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:04 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:04.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:04 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:04 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:04 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:04 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:04.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:04 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.381 INFO:teuthology.orchestra.run.smithi016.stdout:# minimal ceph.conf for 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:17:05.381 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-09-17T12:17:05.381 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:17:05.382 INFO:teuthology.orchestra.run.smithi016.stdout: mon_host = [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] 2024-09-17T12:17:05.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:05 smithi196 ceph-mon[27428]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:17:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:05 smithi196 ceph-mon[27428]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:17:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:05 smithi196 ceph-mon[27428]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:17:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:05 smithi196 ceph-mon[27428]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:17:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:05 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:05 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:05 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:05 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:05 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:17:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:05 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:17:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:05 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:05 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[23719]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:17:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[23719]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:17:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[23719]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:17:05.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[23719]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:17:05.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:17:05.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:17:05.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:05.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[34212]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:17:05.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[34212]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:17:05.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[34212]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:17:05.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[34212]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:17:05.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:05.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:17:05.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:17:05.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:05.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:05 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:06.110 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-17T12:17:06.111 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-17T12:17:06.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T12:17:06.139 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-17T12:17:06.139 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:17:06.207 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-17T12:17:06.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T12:17:06.241 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-17T12:17:06.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:17:06.309 INFO:tasks.cephadm:Adding mgr.y on smithi016 2024-09-17T12:17:06.309 INFO:tasks.cephadm:Adding mgr.x on smithi196 2024-09-17T12:17:06.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi016=y;smithi196=x' 2024-09-17T12:17:06.524 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:06 smithi016 ceph-mon[23719]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T12:17:06.524 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:06 smithi016 ceph-mon[23719]: Reconfiguring daemon mon.a on smithi016 2024-09-17T12:17:06.524 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:06 smithi016 ceph-mon[23719]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:06.524 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:06 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/700830939' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:06.524 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:06 smithi016 ceph-mon[34212]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T12:17:06.524 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:06 smithi016 ceph-mon[34212]: Reconfiguring daemon mon.a on smithi016 2024-09-17T12:17:06.524 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:06 smithi016 ceph-mon[34212]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:06.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:06 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/700830939' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:06.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:06 smithi196 ceph-mon[27428]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T12:17:06.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:06 smithi196 ceph-mon[27428]: Reconfiguring daemon mon.a on smithi016 2024-09-17T12:17:06.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:06 smithi196 ceph-mon[27428]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:06.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:06 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/700830939' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:06.543 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:17:07.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:07 smithi196 ceph-mon[27428]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:07.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:07 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:07.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:07 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:07.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:07 smithi196 ceph-mon[27428]: Reconfiguring mon.c (monmap changed)... 2024-09-17T12:17:07.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:07 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:17:07.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:07 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:17:07.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:07 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:07.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:07 smithi196 ceph-mon[27428]: Reconfiguring daemon mon.c on smithi016 2024-09-17T12:17:07.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[23719]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:07.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:07.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:07.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[23719]: Reconfiguring mon.c (monmap changed)... 2024-09-17T12:17:07.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:17:07.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:17:07.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:07.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[23719]: Reconfiguring daemon mon.c on smithi016 2024-09-17T12:17:07.848 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[34212]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:07.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:07.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:07.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[34212]: Reconfiguring mon.c (monmap changed)... 2024-09-17T12:17:07.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:17:07.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:17:07.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:07.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:07 smithi016 ceph-mon[34212]: Reconfiguring daemon mon.c on smithi016 2024-09-17T12:17:08.229 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mgr update... 2024-09-17T12:17:08.803 DEBUG:teuthology.orchestra.run.smithi196:mgr.x> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.x.service 2024-09-17T12:17:08.805 DEBUG:tasks.cephadm:set 0 configs 2024-09-17T12:17:08.806 INFO:tasks.cephadm:Deploying OSDs... 2024-09-17T12:17:08.806 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-17T12:17:08.806 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T12:17:08.826 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T12:17:08.827 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-09-17T12:17:08.886 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T12:17:08.886 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T12:17:08.886 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-17T12:17:08.886 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T12:17:08.886 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T12:17:08.886 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-17 12:16:56.675666579 +0000 2024-09-17T12:17:08.886 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-17 12:13:56.034544319 +0000 2024-09-17T12:17:08.886 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-17 12:13:56.034544319 +0000 2024-09-17T12:17:08.886 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-17 12:13:56.034544319 +0000 2024-09-17T12:17:08.887 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T12:17:08.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:08 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:08.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:08 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:08.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:08 smithi196 ceph-mon[27428]: Reconfiguring mon.b (monmap changed)... 2024-09-17T12:17:08.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:08 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:17:08.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:08 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:17:08.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:08 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:08.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:08 smithi196 ceph-mon[27428]: Reconfiguring daemon mon.b on smithi196 2024-09-17T12:17:08.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:08 smithi196 ceph-mon[27428]: from='client.14202 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi016=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:17:08.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:08 smithi196 ceph-mon[27428]: Saving service mgr spec with placement smithi016=y;smithi196=x;count:2 2024-09-17T12:17:08.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:08 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-17T12:17:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-17T12:17:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.00012205 s, 4.2 MB/s 2024-09-17T12:17:08.956 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T12:17:09.014 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-09-17T12:17:09.072 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T12:17:09.072 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T12:17:09.072 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-17T12:17:09.072 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T12:17:09.072 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T12:17:09.072 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-17 12:16:56.675666579 +0000 2024-09-17T12:17:09.073 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-17 12:13:56.356539225 +0000 2024-09-17T12:17:09.073 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-17 12:13:56.356539225 +0000 2024-09-17T12:17:09.073 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-17 12:13:56.356539225 +0000 2024-09-17T12:17:09.073 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T12:17:09.137 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-17T12:17:09.137 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-17T12:17:09.137 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.00011547 s, 4.4 MB/s 2024-09-17T12:17:09.138 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T12:17:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[23719]: Reconfiguring mon.b (monmap changed)... 2024-09-17T12:17:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:17:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:17:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:09.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[23719]: Reconfiguring daemon mon.b on smithi196 2024-09-17T12:17:09.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[23719]: from='client.14202 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi016=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:17:09.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[23719]: Saving service mgr spec with placement smithi016=y;smithi196=x;count:2 2024-09-17T12:17:09.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:09.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:09.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:09.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[34212]: Reconfiguring mon.b (monmap changed)... 2024-09-17T12:17:09.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:17:09.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:17:09.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:09.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[34212]: Reconfiguring daemon mon.b on smithi196 2024-09-17T12:17:09.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[34212]: from='client.14202 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi016=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:17:09.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[34212]: Saving service mgr spec with placement smithi016=y;smithi196=x;count:2 2024-09-17T12:17:09.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:08 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:09.196 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-09-17T12:17:09.253 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T12:17:09.254 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T12:17:09.254 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-17T12:17:09.254 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T12:17:09.254 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T12:17:09.254 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-17 12:16:56.675666579 +0000 2024-09-17T12:17:09.254 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-17 12:13:56.690533941 +0000 2024-09-17T12:17:09.254 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-17 12:13:56.690533941 +0000 2024-09-17T12:17:09.254 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-17 12:13:56.690533941 +0000 2024-09-17T12:17:09.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T12:17:09.319 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-17T12:17:09.319 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-17T12:17:09.319 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 9.8983e-05 s, 5.2 MB/s 2024-09-17T12:17:09.320 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T12:17:09.378 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-09-17T12:17:09.434 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T12:17:09.434 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T12:17:09.434 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-17T12:17:09.434 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T12:17:09.434 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T12:17:09.434 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-17 12:16:56.676666563 +0000 2024-09-17T12:17:09.434 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-17 12:13:57.017528768 +0000 2024-09-17T12:17:09.434 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-17 12:13:57.017528768 +0000 2024-09-17T12:17:09.435 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-17 12:13:57.017528768 +0000 2024-09-17T12:17:09.435 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T12:17:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-17T12:17:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-17T12:17:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000119289 s, 4.3 MB/s 2024-09-17T12:17:09.501 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T12:17:09.562 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-17T12:17:09.562 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T12:17:09.580 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T12:17:09.580 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-09-17T12:17:09.638 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T12:17:09.638 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T12:17:09.638 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-17T12:17:09.638 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T12:17:09.638 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T12:17:09.638 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-17 12:16:47.468343549 +0000 2024-09-17T12:17:09.639 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-17 12:13:53.412775738 +0000 2024-09-17T12:17:09.639 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-17 12:13:53.412775738 +0000 2024-09-17T12:17:09.639 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-17 12:13:53.412775738 +0000 2024-09-17T12:17:09.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T12:17:09.703 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-17T12:17:09.703 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-17T12:17:09.703 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000215033 s, 2.4 MB/s 2024-09-17T12:17:09.704 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T12:17:09.760 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-09-17T12:17:09.815 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T12:17:09.816 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T12:17:09.816 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-17T12:17:09.816 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T12:17:09.816 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T12:17:09.816 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-17 12:16:47.468343549 +0000 2024-09-17T12:17:09.816 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-17 12:13:53.734768685 +0000 2024-09-17T12:17:09.816 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-17 12:13:53.734768685 +0000 2024-09-17T12:17:09.816 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-17 12:13:53.734768685 +0000 2024-09-17T12:17:09.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T12:17:09.880 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-17T12:17:09.880 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-17T12:17:09.880 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000181719 s, 2.8 MB/s 2024-09-17T12:17:09.881 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T12:17:09.937 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-09-17T12:17:09.994 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T12:17:09.994 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T12:17:09.994 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-17T12:17:09.994 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T12:17:09.994 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T12:17:09.994 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-17 12:16:47.469343527 +0000 2024-09-17T12:17:09.995 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-17 12:13:54.057761611 +0000 2024-09-17T12:17:09.995 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-17 12:13:54.057761611 +0000 2024-09-17T12:17:09.995 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-17 12:13:54.057761611 +0000 2024-09-17T12:17:09.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T12:17:10.057 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-17T12:17:10.057 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-17T12:17:10.057 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000172128 s, 3.0 MB/s 2024-09-17T12:17:10.060 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T12:17:10.114 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-09-17T12:17:10.169 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T12:17:10.170 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T12:17:10.170 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-17T12:17:10.170 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T12:17:10.170 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T12:17:10.170 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-17 12:16:47.469343527 +0000 2024-09-17T12:17:10.170 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-17 12:13:54.375754646 +0000 2024-09-17T12:17:10.170 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-17 12:13:54.375754646 +0000 2024-09-17T12:17:10.170 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-17 12:13:54.375754646 +0000 2024-09-17T12:17:10.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T12:17:10.234 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-17T12:17:10.235 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-17T12:17:10.235 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000189733 s, 2.7 MB/s 2024-09-17T12:17:10.236 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T12:17:10.297 INFO:tasks.cephadm:Deploying osd.0 on smithi016 with /dev/vg_nvme/lv_4... 2024-09-17T12:17:10.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-17T12:17:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:17:10.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:10 smithi196 ceph-mon[27428]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:10.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:10 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:10.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:10 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:10.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:10 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:10.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:10 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:10.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:10 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:10.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:10 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:10.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:10 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:17:10.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:10 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T12:17:10.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:10 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:17:10.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:10 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:10.628 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[23719]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:10.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:10.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:10.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:10.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:10.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:10.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:10.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:17:10.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T12:17:10.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:17:10.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:10.629 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[34212]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:10.630 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:10.630 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:10.630 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:10.630 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:10.630 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:10.630 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:10.630 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:17:10.630 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T12:17:10.630 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:17:10.630 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:10 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:11.454 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:11 smithi196 ceph-mon[27428]: Deploying daemon mgr.x on smithi196 2024-09-17T12:17:11.454 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:11 smithi196 systemd[1]: Starting Ceph mgr.x for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:17:11.503 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:11 smithi016 ceph-mon[23719]: Deploying daemon mgr.x on smithi196 2024-09-17T12:17:11.503 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:11 smithi016 ceph-mon[34212]: Deploying daemon mgr.x on smithi196 2024-09-17T12:17:11.723 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:11 smithi196 podman[29251]: 2024-09-17 12:17:11.35504963 +0000 UTC m=+0.016004039 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T12:17:11.723 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:11 smithi196 podman[29251]: 2024-09-17 12:17:11.461433074 +0000 UTC m=+0.122387454 container create 842e33f5d7ddbee5f4bef909c44207f5245ce547a3d55913665c74ca752da746 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-17T12:17:11.723 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:11 smithi196 podman[29251]: 2024-09-17 12:17:11.655337413 +0000 UTC m=+0.316291812 container init 842e33f5d7ddbee5f4bef909c44207f5245ce547a3d55913665c74ca752da746 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0) 2024-09-17T12:17:11.723 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:11 smithi196 podman[29251]: 2024-09-17 12:17:11.658213956 +0000 UTC m=+0.319168340 container start 842e33f5d7ddbee5f4bef909c44207f5245ce547a3d55913665c74ca752da746 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b) 2024-09-17T12:17:11.724 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:11 smithi196 ceph-mgr[29270]: -- 172.21.15.196:0/1626246821 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c557e6ad00 con 0x55c557e3f400 2024-09-17T12:17:11.724 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:11 smithi196 bash[29251]: 842e33f5d7ddbee5f4bef909c44207f5245ce547a3d55913665c74ca752da746 2024-09-17T12:17:11.998 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:11 smithi196 systemd[1]: Started Ceph mgr.x for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:17:11.998 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:11 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:11.779+0000 7f2b265c5040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T12:17:11.998 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:11 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:11.837+0000 7f2b265c5040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T12:17:11.998 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:11 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:11.995+0000 7f2b265c5040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T12:17:12.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:12 smithi196 ceph-mon[27428]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:12 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:12 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:12 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:12 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:12 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:12.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:12 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:12.068+0000 7f2b265c5040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T12:17:12.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:12 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:12.237+0000 7f2b265c5040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:17:12.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:12 smithi016 ceph-mon[23719]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:12.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:12 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:12.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:12 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:12.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:12 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:12.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:12 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:12.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:12 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:12.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:12 smithi016 ceph-mon[34212]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:12.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:12 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:12.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:12 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:12.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:12 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:12.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:12 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:12.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:12 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:12.702 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:12 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:12.378+0000 7f2b265c5040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T12:17:12.702 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:12 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:12.699+0000 7f2b265c5040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T12:17:12.814 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:17:12.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2024-09-17T12:17:12.956 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:12 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:12.760+0000 7f2b265c5040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T12:17:13.053 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:17:13.276 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:12 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:12.954+0000 7f2b265c5040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T12:17:14.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[23719]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:14.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:14.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:14.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:14.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:14.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:14.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[23719]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T12:17:14.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:17:14.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:17:14.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:14.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[23719]: Reconfiguring daemon mgr.y on smithi016 2024-09-17T12:17:14.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[34212]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:14.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:14.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:14.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:14.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:14.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:14.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[34212]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T12:17:14.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:17:14.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:17:14.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:14.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:13 smithi016 ceph-mon[34212]: Reconfiguring daemon mgr.y on smithi016 2024-09-17T12:17:14.276 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:13 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:13.820+0000 7f2b265c5040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T12:17:14.276 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:13 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:13.886+0000 7f2b265c5040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T12:17:14.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:13 smithi196 ceph-mon[27428]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:14.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:13 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:14.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:13 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:14.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:13 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:14.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:13 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:14.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:13 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:14.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:13 smithi196 ceph-mon[27428]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T12:17:14.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:13 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:17:14.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:13 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:17:14.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:13 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:14.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:13 smithi196 ceph-mon[27428]: Reconfiguring daemon mgr.y on smithi016 2024-09-17T12:17:14.870 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:14 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:14.595+0000 7f2b265c5040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T12:17:14.870 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:14 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:14.714+0000 7f2b265c5040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T12:17:14.870 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:14 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:14.868+0000 7f2b265c5040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:17:15.142 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:14 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:14.929+0000 7f2b265c5040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T12:17:15.142 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:15.074+0000 7f2b265c5040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T12:17:15.142 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:15.140+0000 7f2b265c5040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T12:17:15.430 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:15.199+0000 7f2b265c5040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T12:17:15.430 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:15.361+0000 7f2b265c5040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T12:17:15.430 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:15.428+0000 7f2b265c5040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T12:17:15.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[23719]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:15.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:15.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:15.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:15.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:15.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:15.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:15.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:15.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[34212]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:15.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:15.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:15.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:15.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:15.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:15.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:15.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:15 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:15.697 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-mon[27428]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:15.697 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:15.697 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:15.697 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:15.697 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:15.697 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:15.697 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:15.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:15.698 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:15.497+0000 7f2b265c5040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T12:17:15.698 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:15.580+0000 7f2b265c5040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T12:17:15.698 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:15.637+0000 7f2b265c5040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T12:17:15.698 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:15.695+0000 7f2b265c5040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T12:17:16.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:16 smithi196 ceph-mon[27428]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:17:16.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:16 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:17:16.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:16 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:17:16.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:16 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:16.777 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:17:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:17:16.467+0000 7f2b265c5040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T12:17:16.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:16 smithi016 ceph-mon[23719]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:17:16.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:16 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:17:16.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:16 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:17:16.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:16 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:16.826 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:16 smithi016 ceph-mon[34212]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:17:16.826 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:16 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:17:16.826 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:16 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:17:16.827 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:16 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:17.640 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:17 smithi016 ceph-mon[34212]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:17.640 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:17 smithi016 ceph-mon[34212]: Standby manager daemon x started 2024-09-17T12:17:17.640 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:17 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/1849233124' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:17:17.640 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:17 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/1849233124' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:17:17.640 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:17 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/1849233124' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:17:17.640 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:17 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/1849233124' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:17:17.640 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:17 smithi016 ceph-mon[23719]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:17.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:17 smithi016 ceph-mon[23719]: Standby manager daemon x started 2024-09-17T12:17:17.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:17 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/1849233124' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:17:17.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:17 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/1849233124' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:17:17.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:17 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/1849233124' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:17:17.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:17 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/1849233124' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:17:17.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:17 smithi196 ceph-mon[27428]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:17.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:17 smithi196 ceph-mon[27428]: Standby manager daemon x started 2024-09-17T12:17:17.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:17 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/1849233124' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:17:17.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:17 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/1849233124' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:17:17.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:17 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/1849233124' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:17:17.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:17 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/1849233124' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:17:18.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:18 smithi196 ceph-mon[27428]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-17T12:17:18.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:18 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:17:18.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:18 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2744031238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32f1f69d-aaf7-42ed-92f0-9d1aa8bdd3c4"}]: dispatch 2024-09-17T12:17:18.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:18 smithi196 ceph-mon[27428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32f1f69d-aaf7-42ed-92f0-9d1aa8bdd3c4"}]: dispatch 2024-09-17T12:17:18.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:18 smithi196 ceph-mon[27428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32f1f69d-aaf7-42ed-92f0-9d1aa8bdd3c4"}]': finished 2024-09-17T12:17:18.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:18 smithi196 ceph-mon[27428]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T12:17:18.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:18 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:18.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[23719]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-17T12:17:18.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:17:18.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2744031238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32f1f69d-aaf7-42ed-92f0-9d1aa8bdd3c4"}]: dispatch 2024-09-17T12:17:18.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[23719]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32f1f69d-aaf7-42ed-92f0-9d1aa8bdd3c4"}]: dispatch 2024-09-17T12:17:18.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[23719]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32f1f69d-aaf7-42ed-92f0-9d1aa8bdd3c4"}]': finished 2024-09-17T12:17:18.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[23719]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T12:17:18.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:18.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[34212]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-17T12:17:18.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:17:18.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2744031238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32f1f69d-aaf7-42ed-92f0-9d1aa8bdd3c4"}]: dispatch 2024-09-17T12:17:18.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[34212]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32f1f69d-aaf7-42ed-92f0-9d1aa8bdd3c4"}]: dispatch 2024-09-17T12:17:18.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[34212]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32f1f69d-aaf7-42ed-92f0-9d1aa8bdd3c4"}]': finished 2024-09-17T12:17:18.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[34212]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T12:17:18.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:18 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:19.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:19 smithi196 ceph-mon[27428]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:19.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:19 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1263446684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:17:19.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:19 smithi016 ceph-mon[23719]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:19.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:19 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1263446684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:17:19.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:19 smithi016 ceph-mon[34212]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:19.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:19 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1263446684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:17:21.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:21 smithi016 ceph-mon[23719]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:21.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:21 smithi016 ceph-mon[34212]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:21.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:21 smithi196 ceph-mon[27428]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:23.619 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:23 smithi016 ceph-mon[23719]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:23.620 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:23 smithi016 ceph-mon[34212]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:23.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:23 smithi196 ceph-mon[27428]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:24.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T12:17:24.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:24.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:24 smithi016 ceph-mon[23719]: Deploying daemon osd.0 on smithi016 2024-09-17T12:17:24.582 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T12:17:24.582 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:24.582 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:24 smithi016 ceph-mon[34212]: Deploying daemon osd.0 on smithi016 2024-09-17T12:17:24.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T12:17:24.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:24.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:24 smithi196 ceph-mon[27428]: Deploying daemon osd.0 on smithi016 2024-09-17T12:17:25.721 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:25 smithi016 ceph-mon[23719]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:25.721 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:25 smithi016 ceph-mon[34212]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:25.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:25 smithi196 ceph-mon[27428]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:27.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:27 smithi196 ceph-mon[27428]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:27.804 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:27 smithi016 ceph-mon[23719]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:27.805 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:27 smithi016 ceph-mon[34212]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:29.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:29 smithi016 ceph-mon[23719]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:29.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:29 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:29.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:29 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:29.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:29 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:29.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:29 smithi016 ceph-mon[34212]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:29.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:29 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:29.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:29 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:29.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:29 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:29.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:29 smithi196 ceph-mon[27428]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:29 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:29 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:29 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:31.633 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:31 smithi016 ceph-mon[23719]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:31.634 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:31 smithi016 ceph-mon[34212]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:31.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:31 smithi196 ceph-mon[27428]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:32.294 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 0 on host 'smithi016' 2024-09-17T12:17:32.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:32 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:32 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:32 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:32.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:32 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:32.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:32 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:32 smithi196 ceph-mon[27428]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T12:17:32.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:32 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:32.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:32 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:32 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.931 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.931 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.932 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:32.932 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:32.932 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.932 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[23719]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T12:17:32.932 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:32.932 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.932 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.932 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.932 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.932 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:32.933 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:32.933 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.933 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[34212]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T12:17:32.933 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:32.933 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.933 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:32 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:32.951 DEBUG:teuthology.orchestra.run.smithi016:osd.0> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.0.service 2024-09-17T12:17:32.954 INFO:tasks.cephadm:Deploying osd.1 on smithi016 with /dev/vg_nvme/lv_3... 2024-09-17T12:17:32.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-17T12:17:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:17:34.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:33 smithi016 ceph-mon[23719]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:34.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:33 smithi016 ceph-mon[23719]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T12:17:34.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:33 smithi016 ceph-mon[23719]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T12:17:34.089 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:33 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:34.089 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:33 smithi016 ceph-mon[23719]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:17:34.089 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:33 smithi016 ceph-mon[34212]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:34.089 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:33 smithi016 ceph-mon[34212]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T12:17:34.089 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:33 smithi016 ceph-mon[34212]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T12:17:34.089 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:33 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:34.089 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:33 smithi016 ceph-mon[34212]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:17:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:33 smithi196 ceph-mon[27428]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:33 smithi196 ceph-mon[27428]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T12:17:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:33 smithi196 ceph-mon[27428]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T12:17:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:33 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:33 smithi196 ceph-mon[27428]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:17:34.441 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0[38071]: 2024-09-17T12:17:34.285+0000 7f77c4d14640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T12:17:34.442 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0[38071]: 2024-09-17T12:17:34.288+0000 7f77bcb35640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:17:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-mon[23719]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-17T12:17:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-mon[23719]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T12:17:35.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:35.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:35.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-mon[23719]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' 2024-09-17T12:17:35.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-mon[34212]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-17T12:17:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-mon[34212]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T12:17:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-mon[34212]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' 2024-09-17T12:17:35.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:34 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:35.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:34 smithi196 ceph-mon[27428]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-17T12:17:35.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:34 smithi196 ceph-mon[27428]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T12:17:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:34 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:34 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:34 smithi196 ceph-mon[27428]: from='osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370]' entity='osd.0' 2024-09-17T12:17:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:34 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:36.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:35 smithi196 ceph-mon[27428]: purged_snaps scrub starts 2024-09-17T12:17:36.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:35 smithi196 ceph-mon[27428]: purged_snaps scrub ok 2024-09-17T12:17:36.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:35 smithi196 ceph-mon[27428]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:36.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:35 smithi196 ceph-mon[27428]: osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370] boot 2024-09-17T12:17:36.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:35 smithi196 ceph-mon[27428]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T12:17:36.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:35 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:36.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:35 smithi016 ceph-mon[23719]: purged_snaps scrub starts 2024-09-17T12:17:36.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:35 smithi016 ceph-mon[23719]: purged_snaps scrub ok 2024-09-17T12:17:36.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:35 smithi016 ceph-mon[23719]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:36.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:35 smithi016 ceph-mon[23719]: osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370] boot 2024-09-17T12:17:36.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:35 smithi016 ceph-mon[23719]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T12:17:36.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:35 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:36.296 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:35 smithi016 ceph-mon[34212]: purged_snaps scrub starts 2024-09-17T12:17:36.296 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:35 smithi016 ceph-mon[34212]: purged_snaps scrub ok 2024-09-17T12:17:36.296 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:35 smithi016 ceph-mon[34212]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T12:17:36.296 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:35 smithi016 ceph-mon[34212]: osd.0 [v2:172.21.15.16:6802/869442370,v1:172.21.15.16:6803/869442370] boot 2024-09-17T12:17:36.296 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:35 smithi016 ceph-mon[34212]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T12:17:36.296 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:35 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:17:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:17:37.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[34212]: Detected new or changed devices on smithi016 2024-09-17T12:17:37.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:37.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:37.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:17:37.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[34212]: Adjusting osd_memory_target on smithi016 to 16068M 2024-09-17T12:17:37.590 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:37.590 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:37.590 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:37.590 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:37.590 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[34212]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T12:17:37.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[23719]: Detected new or changed devices on smithi016 2024-09-17T12:17:37.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:37.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:37.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:17:37.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[23719]: Adjusting osd_memory_target on smithi016 to 16068M 2024-09-17T12:17:37.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:37.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:37.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:37.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:37.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:37 smithi016 ceph-mon[23719]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T12:17:37.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2024-09-17T12:17:37.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:37 smithi196 ceph-mon[27428]: Detected new or changed devices on smithi016 2024-09-17T12:17:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:37 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:37 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:37 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:17:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:37 smithi196 ceph-mon[27428]: Adjusting osd_memory_target on smithi016 to 16068M 2024-09-17T12:17:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:37 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:37 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:37 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:37 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:37 smithi196 ceph-mon[27428]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T12:17:37.792 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:17:38.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:38 smithi016 ceph-mon[23719]: pgmap v39: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:38 smithi016 ceph-mon[34212]: pgmap v39: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:38.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:38 smithi196 ceph-mon[27428]: pgmap v39: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:40.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:40 smithi016 ceph-mon[23719]: pgmap v41: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:40.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:40 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:17:40.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:40 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:17:40.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:40 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:40.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:40 smithi016 ceph-mon[34212]: pgmap v41: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:40.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:40 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:17:40.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:40 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:17:40.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:40 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:40.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:40 smithi196 ceph-mon[27428]: pgmap v41: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:40.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:40 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:17:40.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:40 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:17:40.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:40 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:41.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:41 smithi016 ceph-mon[23719]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:17:41.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:41 smithi016 ceph-mon[34212]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:17:41.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:41 smithi196 ceph-mon[27428]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:17:42.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:42 smithi016 ceph-mon[23719]: pgmap v42: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:42.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:42 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/4000911628' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2dc9d4d-bcc8-4492-af56-57caea2ec732"}]: dispatch 2024-09-17T12:17:42.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:42 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/4000911628' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2dc9d4d-bcc8-4492-af56-57caea2ec732"}]': finished 2024-09-17T12:17:42.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:42 smithi016 ceph-mon[23719]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T12:17:42.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:42 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:42.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:42 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2487437980' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:17:42.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:42 smithi016 ceph-mon[34212]: pgmap v42: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:42.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:42 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/4000911628' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2dc9d4d-bcc8-4492-af56-57caea2ec732"}]: dispatch 2024-09-17T12:17:42.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:42 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/4000911628' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2dc9d4d-bcc8-4492-af56-57caea2ec732"}]': finished 2024-09-17T12:17:42.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:42 smithi016 ceph-mon[34212]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T12:17:42.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:42 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:42.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:42 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2487437980' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:17:42.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:42 smithi196 ceph-mon[27428]: pgmap v42: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:42.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:42 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/4000911628' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2dc9d4d-bcc8-4492-af56-57caea2ec732"}]: dispatch 2024-09-17T12:17:42.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:42 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/4000911628' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2dc9d4d-bcc8-4492-af56-57caea2ec732"}]': finished 2024-09-17T12:17:42.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:42 smithi196 ceph-mon[27428]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T12:17:42.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:42 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:42.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:42 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2487437980' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:17:44.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:44 smithi016 ceph-mon[23719]: pgmap v44: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:44.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:44 smithi016 ceph-mon[34212]: pgmap v44: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:44.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:44 smithi196 ceph-mon[27428]: pgmap v44: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:46.643 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:46 smithi016 ceph-mon[23719]: pgmap v45: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:46.644 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:46 smithi016 ceph-mon[34212]: pgmap v45: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:46.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:46 smithi196 ceph-mon[27428]: pgmap v45: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:48.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:48 smithi016 ceph-mon[23719]: pgmap v46: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:48.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T12:17:48.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:48.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:48 smithi016 ceph-mon[34212]: pgmap v46: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:48.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T12:17:48.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:48.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:48 smithi196 ceph-mon[27428]: pgmap v46: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:48.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T12:17:48.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:49.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:49 smithi016 ceph-mon[23719]: Deploying daemon osd.1 on smithi016 2024-09-17T12:17:49.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:49 smithi016 ceph-mon[34212]: Deploying daemon osd.1 on smithi016 2024-09-17T12:17:49.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:49 smithi196 ceph-mon[27428]: Deploying daemon osd.1 on smithi016 2024-09-17T12:17:50.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:50 smithi016 ceph-mon[23719]: pgmap v47: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:50.629 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:50 smithi016 ceph-mon[34212]: pgmap v47: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:50.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:50 smithi196 ceph-mon[27428]: pgmap v47: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:52.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:52 smithi016 ceph-mon[23719]: pgmap v48: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:52.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:52 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:52.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:52 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:52.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:52 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:52.678 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:52 smithi016 ceph-mon[34212]: pgmap v48: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:52.678 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:52 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:52.678 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:52 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:52.678 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:52 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:52.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:52 smithi196 ceph-mon[27428]: pgmap v48: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:52.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:52 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:52.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:52 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:52.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:52 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:54.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:54 smithi016 ceph-mon[23719]: pgmap v49: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:54.576 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:54 smithi016 ceph-mon[34212]: pgmap v49: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:54.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:54 smithi196 ceph-mon[27428]: pgmap v49: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:55.546 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 1 on host 'smithi016' 2024-09-17T12:17:55.771 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[23719]: pgmap v50: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:55.772 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:55.772 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:55.772 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:55.772 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:55.772 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:55.772 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[23719]: from='osd.1 [v2:172.21.15.16:6810/755127111,v1:172.21.15.16:6811/755127111]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T12:17:55.772 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[23719]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T12:17:56.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:55 smithi196 ceph-mon[27428]: pgmap v50: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:56.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:55 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:56.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:55 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:56.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:55 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:56.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:55 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:56.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:55 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:56.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:55 smithi196 ceph-mon[27428]: from='osd.1 [v2:172.21.15.16:6810/755127111,v1:172.21.15.16:6811/755127111]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T12:17:56.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:55 smithi196 ceph-mon[27428]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T12:17:56.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:55 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:56.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:55 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:56.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:55 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:56.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:56.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:56.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:56.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[34212]: pgmap v50: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:56.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:56.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:56.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:17:56.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:17:56.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:56.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[34212]: from='osd.1 [v2:172.21.15.16:6810/755127111,v1:172.21.15.16:6811/755127111]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T12:17:56.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[34212]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T12:17:56.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:17:56.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:56.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:55 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:17:56.214 DEBUG:teuthology.orchestra.run.smithi016:osd.1> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.1.service 2024-09-17T12:17:56.217 INFO:tasks.cephadm:Deploying osd.2 on smithi016 with /dev/vg_nvme/lv_2... 2024-09-17T12:17:56.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-17T12:17:57.280 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:17:57.524 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:57 smithi016 ceph-mon[23719]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T12:17:57.524 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:57 smithi016 ceph-mon[23719]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T12:17:57.524 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:57 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:57.524 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:57 smithi016 ceph-mon[23719]: from='osd.1 [v2:172.21.15.16:6810/755127111,v1:172.21.15.16:6811/755127111]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:17:57.525 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:57 smithi016 ceph-mon[23719]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:17:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:57 smithi016 ceph-mon[34212]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T12:17:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:57 smithi016 ceph-mon[34212]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T12:17:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:57 smithi016 ceph-mon[34212]: from='osd.1 [v2:172.21.15.16:6810/755127111,v1:172.21.15.16:6811/755127111]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:17:57.525 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:57 smithi016 ceph-mon[34212]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:17:57.525 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:17:57 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1[44110]: 2024-09-17T12:17:57.385+0000 7f5609852640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T12:17:57.526 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:17:57 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1[44110]: 2024-09-17T12:17:57.387+0000 7f560567b640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:17:57.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:57 smithi196 ceph-mon[27428]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T12:17:57.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:57 smithi196 ceph-mon[27428]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T12:17:57.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:57 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:57.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:57 smithi196 ceph-mon[27428]: from='osd.1 [v2:172.21.15.16:6810/755127111,v1:172.21.15.16:6811/755127111]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:17:57.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:57 smithi196 ceph-mon[27428]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:17:58.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:58 smithi016 ceph-mon[34212]: pgmap v52: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:58.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:58 smithi016 ceph-mon[34212]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-17T12:17:58.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:58 smithi016 ceph-mon[34212]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T12:17:58.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:58 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:58.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:58 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:58.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:58 smithi016 ceph-mon[23719]: pgmap v52: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:58.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:58 smithi016 ceph-mon[23719]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-17T12:17:58.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:58 smithi016 ceph-mon[23719]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T12:17:58.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:58 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:58.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:58 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:58.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:58 smithi196 ceph-mon[27428]: pgmap v52: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:17:58.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:58 smithi196 ceph-mon[27428]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-17T12:17:58.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:58 smithi196 ceph-mon[27428]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T12:17:58.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:58 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:58.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:58 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:59.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:59 smithi016 ceph-mon[23719]: purged_snaps scrub starts 2024-09-17T12:17:59.395 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:59 smithi016 ceph-mon[34212]: purged_snaps scrub starts 2024-09-17T12:17:59.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:59 smithi016 ceph-mon[23719]: purged_snaps scrub ok 2024-09-17T12:17:59.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:59 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:59.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:59 smithi016 ceph-mon[23719]: osd.1 [v2:172.21.15.16:6810/755127111,v1:172.21.15.16:6811/755127111] boot 2024-09-17T12:17:59.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:59 smithi016 ceph-mon[23719]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T12:17:59.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:17:59 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:59.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:59 smithi016 ceph-mon[34212]: purged_snaps scrub ok 2024-09-17T12:17:59.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:59 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:59.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:59 smithi016 ceph-mon[34212]: osd.1 [v2:172.21.15.16:6810/755127111,v1:172.21.15.16:6811/755127111] boot 2024-09-17T12:17:59.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:59 smithi016 ceph-mon[34212]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T12:17:59.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:17:59 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:59.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:59 smithi196 ceph-mon[27428]: purged_snaps scrub starts 2024-09-17T12:17:59.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:59 smithi196 ceph-mon[27428]: purged_snaps scrub ok 2024-09-17T12:17:59.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:59 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:17:59.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:59 smithi196 ceph-mon[27428]: osd.1 [v2:172.21.15.16:6810/755127111,v1:172.21.15.16:6811/755127111] boot 2024-09-17T12:17:59.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:59 smithi196 ceph-mon[27428]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T12:17:59.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:17:59 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:18:00.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[23719]: pgmap v55: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:18:00.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[23719]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T12:18:00.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:00.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:00.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:00.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:00.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:00.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:00.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:00.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:00.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[34212]: pgmap v55: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:18:00.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[34212]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T12:18:00.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:00.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:00.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:00.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:00.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:00.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:00.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:00.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:00 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:00.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:00 smithi196 ceph-mon[27428]: pgmap v55: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-09-17T12:18:00.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:00 smithi196 ceph-mon[27428]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T12:18:00.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:00 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:00.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:00 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:00.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:00 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:00.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:00 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:00.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:00 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:00.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:00 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:00.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:00 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:00.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:00 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:00.971 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:18:00.993 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2024-09-17T12:18:01.202 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:18:01.671 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:01 smithi016 ceph-mon[23719]: Detected new or changed devices on smithi016 2024-09-17T12:18:01.671 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:01 smithi016 ceph-mon[23719]: Adjusting osd_memory_target on smithi016 to 8034M 2024-09-17T12:18:01.671 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:01 smithi016 ceph-mon[34212]: Detected new or changed devices on smithi016 2024-09-17T12:18:01.671 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:01 smithi016 ceph-mon[34212]: Adjusting osd_memory_target on smithi016 to 8034M 2024-09-17T12:18:01.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:01 smithi196 ceph-mon[27428]: Detected new or changed devices on smithi016 2024-09-17T12:18:01.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:01 smithi196 ceph-mon[27428]: Adjusting osd_memory_target on smithi016 to 8034M 2024-09-17T12:18:02.587 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:02 smithi016 ceph-mon[23719]: pgmap v57: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:02.587 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:02 smithi016 ceph-mon[34212]: pgmap v57: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:02.678 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:02 smithi196 ceph-mon[27428]: pgmap v57: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:03.680 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:03 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:18:03.680 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:03 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:18:03.680 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:03 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:03.680 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:03 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:18:03.681 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:03 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:18:03.681 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:03 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:03.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:03 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:18:03.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:03 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:18:03.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:03 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:04.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:04 smithi016 ceph-mon[23719]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:04.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:04 smithi016 ceph-mon[23719]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:18:04.602 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:04 smithi016 ceph-mon[34212]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:04.602 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:04 smithi016 ceph-mon[34212]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:18:04.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:04 smithi196 ceph-mon[27428]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:04.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:04 smithi196 ceph-mon[27428]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:18:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:05 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/561687875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa7dc4a4-7b6f-440c-bdd8-78f7f1786434"}]: dispatch 2024-09-17T12:18:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:05 smithi016 ceph-mon[23719]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa7dc4a4-7b6f-440c-bdd8-78f7f1786434"}]: dispatch 2024-09-17T12:18:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:05 smithi016 ceph-mon[23719]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa7dc4a4-7b6f-440c-bdd8-78f7f1786434"}]': finished 2024-09-17T12:18:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:05 smithi016 ceph-mon[23719]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T12:18:05.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:05 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:05.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:05 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/561687875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa7dc4a4-7b6f-440c-bdd8-78f7f1786434"}]: dispatch 2024-09-17T12:18:05.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:05 smithi016 ceph-mon[34212]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa7dc4a4-7b6f-440c-bdd8-78f7f1786434"}]: dispatch 2024-09-17T12:18:05.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:05 smithi016 ceph-mon[34212]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa7dc4a4-7b6f-440c-bdd8-78f7f1786434"}]': finished 2024-09-17T12:18:05.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:05 smithi016 ceph-mon[34212]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T12:18:05.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:05 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:05.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:05 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/561687875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa7dc4a4-7b6f-440c-bdd8-78f7f1786434"}]: dispatch 2024-09-17T12:18:05.939 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:05 smithi196 ceph-mon[27428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa7dc4a4-7b6f-440c-bdd8-78f7f1786434"}]: dispatch 2024-09-17T12:18:05.939 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:05 smithi196 ceph-mon[27428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa7dc4a4-7b6f-440c-bdd8-78f7f1786434"}]': finished 2024-09-17T12:18:05.939 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:05 smithi196 ceph-mon[27428]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T12:18:05.939 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:05 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:06.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:06 smithi016 ceph-mon[23719]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:06.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:06 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2249261596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:18:06.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:06 smithi016 ceph-mon[34212]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:06.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:06 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2249261596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:18:06.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:06 smithi196 ceph-mon[27428]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:06.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:06 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2249261596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:18:07.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:07 smithi016 ceph-mon[23719]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:07.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:07 smithi016 ceph-mon[34212]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:07.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:07 smithi196 ceph-mon[27428]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:09.699 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:09 smithi016 ceph-mon[23719]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:09.700 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:09 smithi016 ceph-mon[34212]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:09.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:09 smithi196 ceph-mon[27428]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:11.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:11 smithi016 ceph-mon[23719]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:11.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:11 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T12:18:11.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:11 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:11.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:11 smithi016 ceph-mon[23719]: Deploying daemon osd.2 on smithi016 2024-09-17T12:18:11.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:11 smithi016 ceph-mon[34212]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:11.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:11 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T12:18:11.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:11 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:11.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:11 smithi016 ceph-mon[34212]: Deploying daemon osd.2 on smithi016 2024-09-17T12:18:11.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:11 smithi196 ceph-mon[27428]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:11.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:11 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T12:18:11.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:11 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:11.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:11 smithi196 ceph-mon[27428]: Deploying daemon osd.2 on smithi016 2024-09-17T12:18:13.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:13 smithi016 ceph-mon[23719]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:13.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:13 smithi016 ceph-mon[34212]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:13.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:13 smithi196 ceph-mon[27428]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:15.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:15 smithi196 ceph-mon[27428]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:15.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:15 smithi016 ceph-mon[23719]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:15.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:15 smithi016 ceph-mon[34212]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:16.488 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:16 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:18:16.488 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:16 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:16.489 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:16 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:16.489 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:16 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:18:16.489 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:16 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:16.489 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:16 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:16.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:16 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:18:16.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:16 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:16.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:16 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:18.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:18 smithi016 ceph-mon[23719]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:18.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:18 smithi016 ceph-mon[34212]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:18.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:18 smithi196 ceph-mon[27428]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:19.703 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 2 on host 'smithi016' 2024-09-17T12:18:20.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:20 smithi196 ceph-mon[27428]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:20 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:20 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:20 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:20 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:20 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:20 smithi196 ceph-mon[27428]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T12:18:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:20 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:18:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:20 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:20 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.284 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[34212]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:20.284 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.284 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.284 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:20.284 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:20.284 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.284 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[34212]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T12:18:20.285 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:18:20.285 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.285 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[23719]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:20.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:20.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:20.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[23719]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T12:18:20.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:18:20.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:20 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:20.305 DEBUG:teuthology.orchestra.run.smithi016:osd.2> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.2.service 2024-09-17T12:18:20.307 INFO:tasks.cephadm:Deploying osd.3 on smithi016 with /dev/vg_nvme/lv_1... 2024-09-17T12:18:20.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-17T12:18:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:21 smithi016 ceph-mon[34212]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T12:18:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:21 smithi016 ceph-mon[34212]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T12:18:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:21 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:21 smithi016 ceph-mon[34212]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:18:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:21 smithi016 ceph-mon[23719]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T12:18:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:21 smithi016 ceph-mon[23719]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T12:18:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:21 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:21 smithi016 ceph-mon[23719]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:18:21.733 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:18:21 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2[50096]: 2024-09-17T12:18:21.674+0000 7fc116046640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T12:18:21.733 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:18:21 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2[50096]: 2024-09-17T12:18:21.678+0000 7fc10de67640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:18:21.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:21 smithi196 ceph-mon[27428]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T12:18:21.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:21 smithi196 ceph-mon[27428]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T12:18:21.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:21 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:21.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:21 smithi196 ceph-mon[27428]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:18:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:18:22.630 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:22 smithi016 ceph-mon[23719]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:22.630 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:22 smithi016 ceph-mon[23719]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-17T12:18:22.630 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:22 smithi016 ceph-mon[23719]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T12:18:22.631 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:22 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:22.631 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:22 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:22.631 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:22 smithi016 ceph-mon[23719]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' 2024-09-17T12:18:22.631 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:22 smithi016 ceph-mon[34212]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:22.631 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:22 smithi016 ceph-mon[34212]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-17T12:18:22.631 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:22 smithi016 ceph-mon[34212]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T12:18:22.631 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:22 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:22.631 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:22 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:22.631 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:22 smithi016 ceph-mon[34212]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' 2024-09-17T12:18:22.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:22 smithi196 ceph-mon[27428]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:22.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:22 smithi196 ceph-mon[27428]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-17T12:18:22.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:22 smithi196 ceph-mon[27428]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T12:18:22.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:22 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:22.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:22 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:22.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:22 smithi196 ceph-mon[27428]: from='osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493]' entity='osd.2' 2024-09-17T12:18:23.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: purged_snaps scrub starts 2024-09-17T12:18:23.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: purged_snaps scrub ok 2024-09-17T12:18:23.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:23.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493] boot 2024-09-17T12:18:23.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T12:18:23.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:23.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:23.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:23.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:23.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:23.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:23.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:23.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:23.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:23.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:23.607 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: purged_snaps scrub starts 2024-09-17T12:18:23.607 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: purged_snaps scrub ok 2024-09-17T12:18:23.607 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:23.607 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493] boot 2024-09-17T12:18:23.607 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T12:18:23.607 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:23.607 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:23.607 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:23.607 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:23.607 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:23.607 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:23.608 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:23.608 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:23.608 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:23.608 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:23 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:23.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: purged_snaps scrub starts 2024-09-17T12:18:23.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: purged_snaps scrub ok 2024-09-17T12:18:23.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:23.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: osd.2 [v2:172.21.15.16:6818/2313749493,v1:172.21.15.16:6819/2313749493] boot 2024-09-17T12:18:23.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T12:18:23.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:18:23.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:23.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:23.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:23.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:23.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:23.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:23.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:23.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:23.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:23 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:24.653 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:24 smithi016 ceph-mon[23719]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:24.653 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:24 smithi016 ceph-mon[23719]: Detected new or changed devices on smithi016 2024-09-17T12:18:24.654 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:24 smithi016 ceph-mon[23719]: Adjusting osd_memory_target on smithi016 to 5356M 2024-09-17T12:18:24.654 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:24 smithi016 ceph-mon[34212]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:24.654 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:24 smithi016 ceph-mon[34212]: Detected new or changed devices on smithi016 2024-09-17T12:18:24.654 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:24 smithi016 ceph-mon[34212]: Adjusting osd_memory_target on smithi016 to 5356M 2024-09-17T12:18:24.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:24 smithi196 ceph-mon[27428]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T12:18:24.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:24 smithi196 ceph-mon[27428]: Detected new or changed devices on smithi016 2024-09-17T12:18:24.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:24 smithi196 ceph-mon[27428]: Adjusting osd_memory_target on smithi016 to 5356M 2024-09-17T12:18:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:18:24.967 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_1 2024-09-17T12:18:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:18:25.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:25 smithi016 ceph-mon[23719]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T12:18:25.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:25 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-17T12:18:25.590 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:25 smithi016 ceph-mon[34212]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T12:18:25.590 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:25 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-17T12:18:25.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:25 smithi196 ceph-mon[27428]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T12:18:25.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:25 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-17T12:18:26.542 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:26 smithi016 ceph-mon[34212]: pgmap v74: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:26.542 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:26 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-17T12:18:26.542 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:26 smithi016 ceph-mon[34212]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T12:18:26.543 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:26 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T12:18:26.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:26 smithi016 ceph-mon[23719]: pgmap v74: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:26.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:26 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-17T12:18:26.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:26 smithi016 ceph-mon[23719]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T12:18:26.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:26 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T12:18:26.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:26 smithi196 ceph-mon[27428]: pgmap v74: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:26.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:26 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-17T12:18:26.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:26 smithi196 ceph-mon[27428]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T12:18:26.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:26 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T12:18:27.191 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:18:26 smithi016 sudo[54275]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T12:18:27.191 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:18:26 smithi016 sudo[54275]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T12:18:27.191 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:18:26 smithi016 sudo[54275]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T12:18:27.192 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:18:27 smithi016 sudo[54275]: pam_unix(sudo:session): session closed for user root 2024-09-17T12:18:27.650 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:27 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T12:18:27.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:27 smithi016 ceph-mon[23719]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T12:18:27.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:27 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:18:27.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:27 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:18:27.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:27 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:27.651 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:27 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T12:18:27.651 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:27 smithi016 ceph-mon[34212]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T12:18:27.652 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:27 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:18:27.652 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:27 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:18:27.652 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:27 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:27.652 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:18:27 smithi016 sudo[54398]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T12:18:27.652 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:18:27 smithi016 sudo[54398]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T12:18:27.652 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:18:27 smithi016 sudo[54398]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T12:18:27.652 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:18:27 smithi016 sudo[54398]: pam_unix(sudo:session): session closed for user root 2024-09-17T12:18:27.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:27 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T12:18:27.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:27 smithi196 ceph-mon[27428]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T12:18:27.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:27 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:18:27.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:27 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:18:27.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:27 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:28.191 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:18:27 smithi016 sudo[54419]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T12:18:28.191 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:18:27 smithi016 sudo[54419]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T12:18:28.191 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:18:27 smithi016 sudo[54419]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T12:18:28.191 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:18:27 smithi016 sudo[54419]: pam_unix(sudo:session): session closed for user root 2024-09-17T12:18:28.585 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:18:28 smithi016 sudo[54446]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T12:18:28.585 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:18:28 smithi016 sudo[54446]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T12:18:28.585 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:18:28 smithi016 sudo[54446]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T12:18:28.585 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:18:28 smithi016 sudo[54446]: pam_unix(sudo:session): session closed for user root 2024-09-17T12:18:28.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:28 smithi016 ceph-mon[23719]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:28.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:28 smithi016 ceph-mon[23719]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:18:28.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:28 smithi016 ceph-mon[23719]: osdmap e22: 3 total, 3 up, 3 in 2024-09-17T12:18:28.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:28 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:28.586 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:28 smithi016 ceph-mon[34212]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:28.586 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:28 smithi016 ceph-mon[34212]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:18:28.586 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:28 smithi016 ceph-mon[34212]: osdmap e22: 3 total, 3 up, 3 in 2024-09-17T12:18:28.586 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:28 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:28.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:28 smithi196 ceph-mon[27428]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:28.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:28 smithi196 ceph-mon[27428]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:18:28.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:28 smithi196 ceph-mon[27428]: osdmap e22: 3 total, 3 up, 3 in 2024-09-17T12:18:28.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:28 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:28.832 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:18:28 smithi016 sudo[54456]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T12:18:29.191 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:18:28 smithi016 sudo[54456]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T12:18:29.192 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:18:28 smithi016 sudo[54456]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T12:18:29.192 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:18:28 smithi016 sudo[54456]: pam_unix(sudo:session): session closed for user root 2024-09-17T12:18:29.691 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:18:29 smithi016 sudo[54502]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T12:18:29.691 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:18:29 smithi016 sudo[54502]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T12:18:29.691 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:18:29 smithi016 sudo[54502]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T12:18:29.691 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:18:29 smithi016 sudo[54502]: pam_unix(sudo:session): session closed for user root 2024-09-17T12:18:30.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:29 smithi016 sudo[54539]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T12:18:30.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:29 smithi016 sudo[54539]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T12:18:30.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:29 smithi016 sudo[54539]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T12:18:30.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:29 smithi016 sudo[54539]: pam_unix(sudo:session): session closed for user root 2024-09-17T12:18:30.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:30 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3430137883' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81f6d250-32a6-4276-b6bc-aac7e27e1a6f"}]: dispatch 2024-09-17T12:18:30.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:30 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3430137883' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81f6d250-32a6-4276-b6bc-aac7e27e1a6f"}]': finished 2024-09-17T12:18:30.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:30 smithi196 ceph-mon[27428]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T12:18:30.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:30 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:30.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:30 smithi196 ceph-mon[27428]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T12:18:30.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:30 smithi196 sudo[29910]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T12:18:30.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:30 smithi196 sudo[29910]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T12:18:30.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:30 smithi196 sudo[29910]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T12:18:30.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:30 smithi196 sudo[29910]: pam_unix(sudo:session): session closed for user root 2024-09-17T12:18:30.554 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:30 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3430137883' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81f6d250-32a6-4276-b6bc-aac7e27e1a6f"}]: dispatch 2024-09-17T12:18:30.554 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:30 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3430137883' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81f6d250-32a6-4276-b6bc-aac7e27e1a6f"}]': finished 2024-09-17T12:18:30.554 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:30 smithi016 ceph-mon[34212]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T12:18:30.554 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:30 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:30.554 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:30 smithi016 ceph-mon[34212]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T12:18:30.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:30 smithi016 sudo[54597]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T12:18:30.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:30 smithi016 sudo[54597]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T12:18:30.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:30 smithi016 sudo[54597]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T12:18:30.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:30 smithi016 sudo[54597]: pam_unix(sudo:session): session closed for user root 2024-09-17T12:18:30.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:30 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3430137883' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81f6d250-32a6-4276-b6bc-aac7e27e1a6f"}]: dispatch 2024-09-17T12:18:30.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:30 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3430137883' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81f6d250-32a6-4276-b6bc-aac7e27e1a6f"}]': finished 2024-09-17T12:18:30.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:30 smithi016 ceph-mon[23719]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T12:18:30.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:30.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:30 smithi016 ceph-mon[23719]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T12:18:30.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:30 smithi196 sudo[29913]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T12:18:30.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:30 smithi196 sudo[29913]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T12:18:30.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:30 smithi196 sudo[29913]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T12:18:30.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:30 smithi196 sudo[29913]: pam_unix(sudo:session): session closed for user root 2024-09-17T12:18:31.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 sudo[54969]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T12:18:31.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 sudo[54969]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T12:18:31.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 sudo[54969]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T12:18:31.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 sudo[54969]: pam_unix(sudo:session): session closed for user root 2024-09-17T12:18:31.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 sudo[55346]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T12:18:31.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 sudo[55346]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T12:18:31.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 sudo[55346]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T12:18:31.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 sudo[55346]: pam_unix(sudo:session): session closed for user root 2024-09-17T12:18:31.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:31.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/4183976858' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:18:31.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T12:18:31.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:18:31.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:18:31.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:18:31.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T12:18:31.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T12:18:31.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:18:31.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:18:31.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:18:31.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T12:18:31.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:31.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:18:31.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:18:31.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:18:31.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:31.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/4183976858' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:18:31.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T12:18:31.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:18:31.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:18:31.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:18:31.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T12:18:31.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T12:18:31.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:18:31.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:18:31.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:18:31.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T12:18:31.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:31.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:18:31.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:18:31.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:18:32.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:32.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/4183976858' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:18:32.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T12:18:32.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:18:32.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:18:32.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:18:32.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T12:18:32.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T12:18:32.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:18:32.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:18:32.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:18:32.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T12:18:32.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:32.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:18:32.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:18:32.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:18:32.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:32 smithi016 ceph-mon[23719]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:32.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:32 smithi016 ceph-mon[23719]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T12:18:32.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:32 smithi016 ceph-mon[23719]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T12:18:32.730 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:32 smithi016 ceph-mon[34212]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:32.730 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:32 smithi016 ceph-mon[34212]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T12:18:32.730 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:32 smithi016 ceph-mon[34212]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T12:18:32.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:32 smithi196 ceph-mon[27428]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:32.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:32 smithi196 ceph-mon[27428]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T12:18:32.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:32 smithi196 ceph-mon[27428]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T12:18:33.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:33 smithi016 ceph-mon[34212]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:33.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:33 smithi016 ceph-mon[23719]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:34.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:33 smithi196 ceph-mon[27428]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:35.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:35 smithi016 ceph-mon[23719]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-09-17T12:18:35.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:35 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T12:18:35.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:35 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:35.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:35 smithi016 ceph-mon[23719]: Deploying daemon osd.3 on smithi016 2024-09-17T12:18:35.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:35 smithi016 ceph-mon[34212]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-09-17T12:18:35.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:35 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T12:18:35.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:35 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:35.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:35 smithi016 ceph-mon[34212]: Deploying daemon osd.3 on smithi016 2024-09-17T12:18:36.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:35 smithi196 ceph-mon[27428]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-09-17T12:18:36.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:35 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T12:18:36.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:35 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:36.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:35 smithi196 ceph-mon[27428]: Deploying daemon osd.3 on smithi016 2024-09-17T12:18:37.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:37 smithi016 ceph-mon[23719]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-17T12:18:37.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:37 smithi016 ceph-mon[34212]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-17T12:18:38.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:37 smithi196 ceph-mon[27428]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-17T12:18:39.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:39 smithi016 ceph-mon[23719]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-17T12:18:39.743 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:39 smithi016 ceph-mon[34212]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-17T12:18:40.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:39 smithi196 ceph-mon[27428]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-17T12:18:40.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:40 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:18:40.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:40 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:40.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:40 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:40.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:40 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:18:40.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:40 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:40.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:40 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:41.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:40 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:18:41.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:40 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:41.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:40 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:42.352 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:42 smithi016 ceph-mon[34212]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-17T12:18:42.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:42 smithi016 ceph-mon[23719]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-17T12:18:42.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:42 smithi196 ceph-mon[27428]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-17T12:18:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 3 on host 'smithi016' 2024-09-17T12:18:44.433 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:44 smithi016 ceph-mon[23719]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-17T12:18:44.433 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:44 smithi016 ceph-mon[23719]: from='osd.3 [v2:172.21.15.16:6826/3579869089,v1:172.21.15.16:6827/3579869089]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T12:18:44.433 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:44 smithi016 ceph-mon[23719]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T12:18:44.433 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:44 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:44.433 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:44 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:44.433 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:44 smithi016 ceph-mon[34212]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-17T12:18:44.433 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:44 smithi016 ceph-mon[34212]: from='osd.3 [v2:172.21.15.16:6826/3579869089,v1:172.21.15.16:6827/3579869089]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T12:18:44.434 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:44 smithi016 ceph-mon[34212]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T12:18:44.434 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:44 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:44.434 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:44 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:44.457 DEBUG:teuthology.orchestra.run.smithi016:osd.3> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.3.service 2024-09-17T12:18:44.459 INFO:tasks.cephadm:Deploying osd.4 on smithi196 with /dev/vg_nvme/lv_4... 2024-09-17T12:18:44.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-17T12:18:44.663 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:18:44.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:44 smithi196 ceph-mon[27428]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-17T12:18:44.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:44 smithi196 ceph-mon[27428]: from='osd.3 [v2:172.21.15.16:6826/3579869089,v1:172.21.15.16:6827/3579869089]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T12:18:44.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:44 smithi196 ceph-mon[27428]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T12:18:44.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:44 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:44.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:44 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:45.503 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[23719]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T12:18:45.503 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[23719]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T12:18:45.503 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:45.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[23719]: from='osd.3 [v2:172.21.15.16:6826/3579869089,v1:172.21.15.16:6827/3579869089]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:18:45.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[23719]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:18:45.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:45.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:45.505 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[34212]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T12:18:45.505 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[34212]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T12:18:45.505 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:45.505 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[34212]: from='osd.3 [v2:172.21.15.16:6826/3579869089,v1:172.21.15.16:6827/3579869089]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:18:45.505 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[34212]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:18:45.506 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:45.506 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:45.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:45 smithi196 ceph-mon[27428]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T12:18:45.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:45 smithi196 ceph-mon[27428]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T12:18:45.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:45 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:45.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:45 smithi196 ceph-mon[27428]: from='osd.3 [v2:172.21.15.16:6826/3579869089,v1:172.21.15.16:6827/3579869089]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:18:45.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:45 smithi196 ceph-mon[27428]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-17T12:18:45.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:45 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:45.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:45 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:45.857 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3[56177]: 2024-09-17T12:18:45.622+0000 7fd56385d640 -1 osd.3 0 waiting for initial osdmap 2024-09-17T12:18:45.857 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:18:45 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3[56177]: 2024-09-17T12:18:45.626+0000 7fd55ae67640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:18:46.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:46 smithi016 ceph-mon[23719]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:46.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:46 smithi016 ceph-mon[23719]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-17T12:18:46.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:46 smithi016 ceph-mon[23719]: osdmap e25: 4 total, 3 up, 4 in 2024-09-17T12:18:46.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:46 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:46.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:46 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:46.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:46 smithi016 ceph-mon[23719]: from='osd.3 ' entity='osd.3' 2024-09-17T12:18:46.393 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:46 smithi016 ceph-mon[34212]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:46.393 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:46 smithi016 ceph-mon[34212]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-17T12:18:46.393 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:46 smithi016 ceph-mon[34212]: osdmap e25: 4 total, 3 up, 4 in 2024-09-17T12:18:46.394 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:46 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:46.394 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:46 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:46.394 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:46 smithi016 ceph-mon[34212]: from='osd.3 ' entity='osd.3' 2024-09-17T12:18:46.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:46 smithi196 ceph-mon[27428]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:46 smithi196 ceph-mon[27428]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-17T12:18:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:46 smithi196 ceph-mon[27428]: osdmap e25: 4 total, 3 up, 4 in 2024-09-17T12:18:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:46 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:46 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:46 smithi196 ceph-mon[27428]: from='osd.3 ' entity='osd.3' 2024-09-17T12:18:47.034 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-17T12:18:47.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_4 2024-09-17T12:18:47.256 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:18:47.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:47 smithi016 ceph-mon[23719]: purged_snaps scrub starts 2024-09-17T12:18:47.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:47 smithi016 ceph-mon[23719]: purged_snaps scrub ok 2024-09-17T12:18:47.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:47 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:47.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:47 smithi016 ceph-mon[23719]: osd.3 [v2:172.21.15.16:6826/3579869089,v1:172.21.15.16:6827/3579869089] boot 2024-09-17T12:18:47.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:47 smithi016 ceph-mon[23719]: osdmap e26: 4 total, 4 up, 4 in 2024-09-17T12:18:47.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:47 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:47 smithi016 ceph-mon[34212]: purged_snaps scrub starts 2024-09-17T12:18:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:47 smithi016 ceph-mon[34212]: purged_snaps scrub ok 2024-09-17T12:18:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:47 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:47 smithi016 ceph-mon[34212]: osd.3 [v2:172.21.15.16:6826/3579869089,v1:172.21.15.16:6827/3579869089] boot 2024-09-17T12:18:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:47 smithi016 ceph-mon[34212]: osdmap e26: 4 total, 4 up, 4 in 2024-09-17T12:18:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:47 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:47.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:47 smithi196 ceph-mon[27428]: purged_snaps scrub starts 2024-09-17T12:18:47.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:47 smithi196 ceph-mon[27428]: purged_snaps scrub ok 2024-09-17T12:18:47.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:47 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:47.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:47 smithi196 ceph-mon[27428]: osd.3 [v2:172.21.15.16:6826/3579869089,v1:172.21.15.16:6827/3579869089] boot 2024-09-17T12:18:47.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:47 smithi196 ceph-mon[27428]: osdmap e26: 4 total, 4 up, 4 in 2024-09-17T12:18:47.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:47 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:18:48.631 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:48.631 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.631 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.631 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:48.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:48.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:48.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:48.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:48.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:48.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:18:48.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:48.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:48.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:48 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:48.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:48.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:48.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:48.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:48.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:48.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:48.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:18:48.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:48.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:48.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T12:18:48.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:48.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:48.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:48.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:18:48.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:48.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:48.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:48.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:18:48.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:48.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:18:48.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:48 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:18:49.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:49 smithi196 ceph-mon[27428]: Detected new or changed devices on smithi016 2024-09-17T12:18:49.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:49 smithi196 ceph-mon[27428]: Adjusting osd_memory_target on smithi016 to 4017M 2024-09-17T12:18:49.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:49 smithi196 ceph-mon[27428]: osdmap e27: 4 total, 4 up, 4 in 2024-09-17T12:18:49.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:49 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:18:49.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:49 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:18:49.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:49 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:49.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:49 smithi016 ceph-mon[23719]: Detected new or changed devices on smithi016 2024-09-17T12:18:49.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:49 smithi016 ceph-mon[23719]: Adjusting osd_memory_target on smithi016 to 4017M 2024-09-17T12:18:49.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:49 smithi016 ceph-mon[23719]: osdmap e27: 4 total, 4 up, 4 in 2024-09-17T12:18:49.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:49 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:18:49.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:49 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:18:49.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:49 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:49.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:49 smithi016 ceph-mon[34212]: Detected new or changed devices on smithi016 2024-09-17T12:18:49.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:49 smithi016 ceph-mon[34212]: Adjusting osd_memory_target on smithi016 to 4017M 2024-09-17T12:18:49.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:49 smithi016 ceph-mon[34212]: osdmap e27: 4 total, 4 up, 4 in 2024-09-17T12:18:49.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:49 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:18:49.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:49 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:18:49.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:49 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:50.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:50 smithi196 ceph-mon[27428]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:50.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:50 smithi196 ceph-mon[27428]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:18:50.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:50 smithi016 ceph-mon[23719]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:50.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:50 smithi016 ceph-mon[23719]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:18:50.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:50 smithi016 ceph-mon[34212]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:50.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:50 smithi016 ceph-mon[34212]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:18:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[23719]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[23719]: from='client.? 172.21.15.196:0/3605513377' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6198e3b-3a1d-40a0-b3e8-9d34fc42ac23"}]: dispatch 2024-09-17T12:18:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[23719]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6198e3b-3a1d-40a0-b3e8-9d34fc42ac23"}]: dispatch 2024-09-17T12:18:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[23719]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6198e3b-3a1d-40a0-b3e8-9d34fc42ac23"}]': finished 2024-09-17T12:18:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[23719]: osdmap e28: 5 total, 4 up, 5 in 2024-09-17T12:18:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:18:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[23719]: from='client.? 172.21.15.196:0/658280259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:18:52.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[34212]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:52.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[34212]: from='client.? 172.21.15.196:0/3605513377' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6198e3b-3a1d-40a0-b3e8-9d34fc42ac23"}]: dispatch 2024-09-17T12:18:52.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[34212]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6198e3b-3a1d-40a0-b3e8-9d34fc42ac23"}]: dispatch 2024-09-17T12:18:52.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[34212]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6198e3b-3a1d-40a0-b3e8-9d34fc42ac23"}]': finished 2024-09-17T12:18:52.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[34212]: osdmap e28: 5 total, 4 up, 5 in 2024-09-17T12:18:52.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:18:52.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:52 smithi016 ceph-mon[34212]: from='client.? 172.21.15.196:0/658280259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:18:52.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:52 smithi196 ceph-mon[27428]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:52 smithi196 ceph-mon[27428]: from='client.? 172.21.15.196:0/3605513377' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6198e3b-3a1d-40a0-b3e8-9d34fc42ac23"}]: dispatch 2024-09-17T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:52 smithi196 ceph-mon[27428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6198e3b-3a1d-40a0-b3e8-9d34fc42ac23"}]: dispatch 2024-09-17T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:52 smithi196 ceph-mon[27428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6198e3b-3a1d-40a0-b3e8-9d34fc42ac23"}]': finished 2024-09-17T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:52 smithi196 ceph-mon[27428]: osdmap e28: 5 total, 4 up, 5 in 2024-09-17T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:52 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:52 smithi196 ceph-mon[27428]: from='client.? 172.21.15.196:0/658280259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:18:54.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:54 smithi016 ceph-mon[23719]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:54.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:54 smithi016 ceph-mon[34212]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:54.763 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:54 smithi196 ceph-mon[27428]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:56.681 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:56 smithi196 ceph-mon[27428]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:56.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:56 smithi016 ceph-mon[23719]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:56.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:56 smithi016 ceph-mon[34212]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:57.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:57 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T12:18:57.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:57 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:57.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:57 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T12:18:57.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:57 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:57.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T12:18:57.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:18:58.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:58 smithi196 ceph-mon[27428]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:58.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:18:58 smithi196 ceph-mon[27428]: Deploying daemon osd.4 on smithi196 2024-09-17T12:18:58.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:58 smithi016 ceph-mon[23719]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:58.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:18:58 smithi016 ceph-mon[23719]: Deploying daemon osd.4 on smithi196 2024-09-17T12:18:58.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:58 smithi016 ceph-mon[34212]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:18:58.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:18:58 smithi016 ceph-mon[34212]: Deploying daemon osd.4 on smithi196 2024-09-17T12:19:00.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:00 smithi196 ceph-mon[27428]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:00.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:00 smithi016 ceph-mon[23719]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:00.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:00 smithi016 ceph-mon[34212]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:02.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:02 smithi196 ceph-mon[27428]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:02.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:02.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:02.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:02.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:02 smithi016 ceph-mon[23719]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:02.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:02.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:02.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:02.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:02 smithi016 ceph-mon[34212]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:02.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:02.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:02.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:04.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:04 smithi196 ceph-mon[27428]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:04.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:04 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:04.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:04 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:04.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:04 smithi016 ceph-mon[23719]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:04.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:04 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:04.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:04 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:04.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:04 smithi016 ceph-mon[34212]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:04.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:04 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:04.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:04 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:05.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:05 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:05 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:05 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:05 smithi196 ceph-mon[27428]: from='osd.4 [v2:172.21.15.196:6800/590253655,v1:172.21.15.196:6801/590253655]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T12:19:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:05 smithi196 ceph-mon[27428]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T12:19:05.587 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 4 on host 'smithi196' 2024-09-17T12:19:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:05 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:05 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:05 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:05 smithi016 ceph-mon[23719]: from='osd.4 [v2:172.21.15.196:6800/590253655,v1:172.21.15.196:6801/590253655]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T12:19:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:05 smithi016 ceph-mon[23719]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T12:19:05.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:05 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:05.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:05 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:05.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:05 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:05.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:05 smithi016 ceph-mon[34212]: from='osd.4 [v2:172.21.15.196:6800/590253655,v1:172.21.15.196:6801/590253655]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T12:19:05.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:05 smithi016 ceph-mon[34212]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T12:19:06.202 DEBUG:teuthology.orchestra.run.smithi196:osd.4> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.4.service 2024-09-17T12:19:06.204 INFO:tasks.cephadm:Deploying osd.5 on smithi196 with /dev/vg_nvme/lv_3... 2024-09-17T12:19:06.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-17T12:19:06.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:06 smithi196 ceph-mon[27428]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:06.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:06 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:06.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:06 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:06.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:06 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:06.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:06 smithi196 ceph-mon[27428]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T12:19:06.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:06 smithi196 ceph-mon[27428]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T12:19:06.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:06 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:19:06.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:06 smithi196 ceph-mon[27428]: from='osd.4 [v2:172.21.15.196:6800/590253655,v1:172.21.15.196:6801/590253655]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:06.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:06 smithi196 ceph-mon[27428]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:06.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[23719]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:06.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:06.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:06.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:06.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[23719]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T12:19:06.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[23719]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T12:19:06.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:19:06.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[23719]: from='osd.4 [v2:172.21.15.196:6800/590253655,v1:172.21.15.196:6801/590253655]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:06.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[23719]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:06.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[34212]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:06.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:06.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:06.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:06.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[34212]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T12:19:06.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[34212]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T12:19:06.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:19:06.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[34212]: from='osd.4 [v2:172.21.15.196:6800/590253655,v1:172.21.15.196:6801/590253655]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:06.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:06 smithi016 ceph-mon[34212]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:06.915 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:19:07.026 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:19:06 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[32195]: 2024-09-17T12:19:06.598+0000 7fd4581fa640 -1 osd.4 0 waiting for initial osdmap 2024-09-17T12:19:07.027 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:19:06 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[32195]: 2024-09-17T12:19:06.602+0000 7fd454023640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:19:07.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:07 smithi196 ceph-mon[27428]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:07.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:07 smithi196 ceph-mon[27428]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-17T12:19:07.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:07 smithi196 ceph-mon[27428]: osdmap e30: 5 total, 4 up, 5 in 2024-09-17T12:19:07.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:07 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:19:07.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:07 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:19:07.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:07 smithi016 ceph-mon[23719]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:07.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:07 smithi016 ceph-mon[23719]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-17T12:19:07.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:07 smithi016 ceph-mon[23719]: osdmap e30: 5 total, 4 up, 5 in 2024-09-17T12:19:07.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:07 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:19:07.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:07 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:19:07.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:07 smithi016 ceph-mon[34212]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T12:19:07.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:07 smithi016 ceph-mon[34212]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-17T12:19:07.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:07 smithi016 ceph-mon[34212]: osdmap e30: 5 total, 4 up, 5 in 2024-09-17T12:19:07.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:07 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:19:07.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:07 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:19:08.713 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:08 smithi196 ceph-mon[27428]: purged_snaps scrub starts 2024-09-17T12:19:08.713 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:08 smithi196 ceph-mon[27428]: purged_snaps scrub ok 2024-09-17T12:19:08.713 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:08 smithi196 ceph-mon[27428]: osd.4 [v2:172.21.15.196:6800/590253655,v1:172.21.15.196:6801/590253655] boot 2024-09-17T12:19:08.714 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:08 smithi196 ceph-mon[27428]: osdmap e31: 5 total, 5 up, 5 in 2024-09-17T12:19:08.714 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:08 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:19:08.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:08 smithi016 ceph-mon[23719]: purged_snaps scrub starts 2024-09-17T12:19:08.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:08 smithi016 ceph-mon[23719]: purged_snaps scrub ok 2024-09-17T12:19:08.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:08 smithi016 ceph-mon[23719]: osd.4 [v2:172.21.15.196:6800/590253655,v1:172.21.15.196:6801/590253655] boot 2024-09-17T12:19:08.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:08 smithi016 ceph-mon[23719]: osdmap e31: 5 total, 5 up, 5 in 2024-09-17T12:19:08.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:08 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:19:08.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:08 smithi016 ceph-mon[34212]: purged_snaps scrub starts 2024-09-17T12:19:08.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:08 smithi016 ceph-mon[34212]: purged_snaps scrub ok 2024-09-17T12:19:08.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:08 smithi016 ceph-mon[34212]: osd.4 [v2:172.21.15.196:6800/590253655,v1:172.21.15.196:6801/590253655] boot 2024-09-17T12:19:08.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:08 smithi016 ceph-mon[34212]: osdmap e31: 5 total, 5 up, 5 in 2024-09-17T12:19:08.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:08 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:19:09.691 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:09 smithi196 ceph-mon[27428]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:09.691 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:09 smithi196 ceph-mon[27428]: osdmap e32: 5 total, 5 up, 5 in 2024-09-17T12:19:09.691 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:09 smithi196 ceph-mon[27428]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T12:19:09.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:09 smithi016 ceph-mon[23719]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:09.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:09 smithi016 ceph-mon[23719]: osdmap e32: 5 total, 5 up, 5 in 2024-09-17T12:19:09.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:09 smithi016 ceph-mon[23719]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T12:19:09.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:09 smithi016 ceph-mon[34212]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:09.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:09 smithi016 ceph-mon[34212]: osdmap e32: 5 total, 5 up, 5 in 2024-09-17T12:19:09.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:09 smithi016 ceph-mon[34212]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T12:19:11.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:11 smithi196 ceph-mon[27428]: Detected new or changed devices on smithi196 2024-09-17T12:19:11.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:11 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:11.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:11 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:11.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:11 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:11.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:11 smithi196 ceph-mon[27428]: Adjusting osd_memory_target on smithi196 to 17088M 2024-09-17T12:19:11.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:11 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:11.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:11 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:11.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:11 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:11.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:11 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:11.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:11 smithi196 ceph-mon[27428]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T12:19:11.419 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-17T12:19:11.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_3 2024-09-17T12:19:11.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[23719]: Detected new or changed devices on smithi196 2024-09-17T12:19:11.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:11.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:11.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:11.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[23719]: Adjusting osd_memory_target on smithi196 to 17088M 2024-09-17T12:19:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[23719]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T12:19:11.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[34212]: Detected new or changed devices on smithi196 2024-09-17T12:19:11.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[34212]: Adjusting osd_memory_target on smithi196 to 17088M 2024-09-17T12:19:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:11.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:11.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:11.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:11 smithi016 ceph-mon[34212]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T12:19:11.632 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:19:12.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:12 smithi196 ceph-mon[27428]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:12.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:12 smithi196 ceph-mon[27428]: osdmap e35: 5 total, 5 up, 5 in 2024-09-17T12:19:12.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:12 smithi016 ceph-mon[23719]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:12.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:12 smithi016 ceph-mon[23719]: osdmap e35: 5 total, 5 up, 5 in 2024-09-17T12:19:12.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:12 smithi016 ceph-mon[34212]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:12.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:12 smithi016 ceph-mon[34212]: osdmap e35: 5 total, 5 up, 5 in 2024-09-17T12:19:14.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:14 smithi196 ceph-mon[27428]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:14.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:14 smithi196 ceph-mon[27428]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:19:14.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:14 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:19:14.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:14 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:19:14.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:14 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:14.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:14 smithi016 ceph-mon[23719]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:14.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:14 smithi016 ceph-mon[23719]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:19:14.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:14 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:19:14.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:14 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:19:14.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:14 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:14.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:14 smithi016 ceph-mon[34212]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:14.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:14 smithi016 ceph-mon[34212]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:19:14.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:14 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:19:14.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:14 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:19:14.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:14 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:16.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:16 smithi196 ceph-mon[27428]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-09-17T12:19:16.231 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:16 smithi196 ceph-mon[27428]: from='client.? 172.21.15.196:0/1403159200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7eaa3677-35e9-4ce5-8515-ca7a27286824"}]: dispatch 2024-09-17T12:19:16.231 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:16 smithi196 ceph-mon[27428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7eaa3677-35e9-4ce5-8515-ca7a27286824"}]: dispatch 2024-09-17T12:19:16.231 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:16 smithi196 ceph-mon[27428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7eaa3677-35e9-4ce5-8515-ca7a27286824"}]': finished 2024-09-17T12:19:16.231 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:16 smithi196 ceph-mon[27428]: osdmap e36: 6 total, 5 up, 6 in 2024-09-17T12:19:16.231 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:16 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:16.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:16 smithi016 ceph-mon[23719]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-09-17T12:19:16.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:16 smithi016 ceph-mon[23719]: from='client.? 172.21.15.196:0/1403159200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7eaa3677-35e9-4ce5-8515-ca7a27286824"}]: dispatch 2024-09-17T12:19:16.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:16 smithi016 ceph-mon[23719]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7eaa3677-35e9-4ce5-8515-ca7a27286824"}]: dispatch 2024-09-17T12:19:16.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:16 smithi016 ceph-mon[23719]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7eaa3677-35e9-4ce5-8515-ca7a27286824"}]': finished 2024-09-17T12:19:16.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:16 smithi016 ceph-mon[23719]: osdmap e36: 6 total, 5 up, 6 in 2024-09-17T12:19:16.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:16 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:16.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:16 smithi016 ceph-mon[34212]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-09-17T12:19:16.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:16 smithi016 ceph-mon[34212]: from='client.? 172.21.15.196:0/1403159200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7eaa3677-35e9-4ce5-8515-ca7a27286824"}]: dispatch 2024-09-17T12:19:16.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:16 smithi016 ceph-mon[34212]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7eaa3677-35e9-4ce5-8515-ca7a27286824"}]: dispatch 2024-09-17T12:19:16.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:16 smithi016 ceph-mon[34212]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7eaa3677-35e9-4ce5-8515-ca7a27286824"}]': finished 2024-09-17T12:19:16.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:16 smithi016 ceph-mon[34212]: osdmap e36: 6 total, 5 up, 6 in 2024-09-17T12:19:16.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:16 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:17.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:17 smithi016 ceph-mon[23719]: from='client.? 172.21.15.196:0/1953060688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:19:17.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:17 smithi016 ceph-mon[34212]: from='client.? 172.21.15.196:0/1953060688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:19:17.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:17 smithi196 ceph-mon[27428]: from='client.? 172.21.15.196:0/1953060688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:19:18.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:18 smithi196 ceph-mon[27428]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T12:19:18.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:18 smithi016 ceph-mon[23719]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T12:19:18.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:18 smithi016 ceph-mon[34212]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T12:19:20.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:20 smithi196 ceph-mon[27428]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-17T12:19:20.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:20 smithi016 ceph-mon[23719]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-17T12:19:20.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:20 smithi016 ceph-mon[34212]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-17T12:19:22.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:22 smithi196 ceph-mon[27428]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T12:19:22.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:22 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T12:19:22.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:22 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:22.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:22 smithi016 ceph-mon[23719]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T12:19:22.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:22 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T12:19:22.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:22 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:22 smithi016 ceph-mon[34212]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T12:19:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:22 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T12:19:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:22 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:23.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:23 smithi196 ceph-mon[27428]: Deploying daemon osd.5 on smithi196 2024-09-17T12:19:23.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:23 smithi016 ceph-mon[23719]: Deploying daemon osd.5 on smithi196 2024-09-17T12:19:23.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:23 smithi016 ceph-mon[34212]: Deploying daemon osd.5 on smithi196 2024-09-17T12:19:24.364 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:24 smithi196 ceph-mon[27428]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T12:19:24.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:24 smithi016 ceph-mon[34212]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T12:19:24.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:24 smithi016 ceph-mon[23719]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T12:19:26.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:26 smithi196 ceph-mon[27428]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:26.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:26 smithi016 ceph-mon[23719]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:26.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:26 smithi016 ceph-mon[34212]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:27.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:27 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:27.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:27 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:27.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:27 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:27.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:27 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:27.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:27 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:27.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:27 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:27.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:27 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:27.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:27 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:27.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:27 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:28.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:28 smithi196 ceph-mon[27428]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:28.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:28 smithi016 ceph-mon[23719]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:28.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:28 smithi016 ceph-mon[34212]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:30.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:30 smithi196 ceph-mon[27428]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:30.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:30 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:30.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:30 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:30.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:30 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:30.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:30 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:30.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:30 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:30.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:30 smithi196 ceph-mon[27428]: from='osd.5 [v2:172.21.15.196:6808/2204973118,v1:172.21.15.196:6809/2204973118]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T12:19:30.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:30 smithi196 ceph-mon[27428]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T12:19:30.276 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 5 on host 'smithi196' 2024-09-17T12:19:30.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[23719]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:30.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:30.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:30.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:30.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:30.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:30.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[23719]: from='osd.5 [v2:172.21.15.196:6808/2204973118,v1:172.21.15.196:6809/2204973118]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T12:19:30.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[23719]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T12:19:30.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[34212]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:30.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:30.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:30.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:30.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:30.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:30.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[34212]: from='osd.5 [v2:172.21.15.196:6808/2204973118,v1:172.21.15.196:6809/2204973118]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T12:19:30.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:30 smithi016 ceph-mon[34212]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T12:19:30.950 DEBUG:teuthology.orchestra.run.smithi196:osd.5> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.5.service 2024-09-17T12:19:30.952 INFO:tasks.cephadm:Deploying osd.6 on smithi196 with /dev/vg_nvme/lv_2... 2024-09-17T12:19:30.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-17T12:19:31.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:31.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:31.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:31.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:31 smithi196 ceph-mon[27428]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T12:19:31.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:31 smithi196 ceph-mon[27428]: osdmap e37: 6 total, 5 up, 6 in 2024-09-17T12:19:31.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:31 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:31.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:31 smithi196 ceph-mon[27428]: from='osd.5 [v2:172.21.15.196:6808/2204973118,v1:172.21.15.196:6809/2204973118]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:31.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:31 smithi196 ceph-mon[27428]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:31.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:31.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:31.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:31.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[34212]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T12:19:31.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[34212]: osdmap e37: 6 total, 5 up, 6 in 2024-09-17T12:19:31.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:31.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[34212]: from='osd.5 [v2:172.21.15.196:6808/2204973118,v1:172.21.15.196:6809/2204973118]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:31.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[34212]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:31.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:31.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:31.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:31.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[23719]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T12:19:31.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[23719]: osdmap e37: 6 total, 5 up, 6 in 2024-09-17T12:19:31.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:31.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[23719]: from='osd.5 [v2:172.21.15.196:6808/2204973118,v1:172.21.15.196:6809/2204973118]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:31.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:31 smithi016 ceph-mon[23719]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:31.992 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:19:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[37768]: 2024-09-17T12:19:31.535+0000 7f323b62a640 -1 osd.5 0 waiting for initial osdmap 2024-09-17T12:19:31.992 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:19:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[37768]: 2024-09-17T12:19:31.540+0000 7f3237453640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:19:32.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:32 smithi196 ceph-mon[27428]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:32 smithi196 ceph-mon[27428]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-17T12:19:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:32 smithi196 ceph-mon[27428]: osdmap e38: 6 total, 5 up, 6 in 2024-09-17T12:19:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:32 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:32 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:32.577 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:19:32.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:32 smithi016 ceph-mon[23719]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:32.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:32 smithi016 ceph-mon[23719]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-17T12:19:32.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:32 smithi016 ceph-mon[23719]: osdmap e38: 6 total, 5 up, 6 in 2024-09-17T12:19:32.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:32 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:32.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:32 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:32.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:32 smithi016 ceph-mon[34212]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:32.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:32 smithi016 ceph-mon[34212]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-17T12:19:32.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:32 smithi016 ceph-mon[34212]: osdmap e38: 6 total, 5 up, 6 in 2024-09-17T12:19:32.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:32 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:32.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:32 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:33.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:33 smithi196 ceph-mon[27428]: purged_snaps scrub starts 2024-09-17T12:19:33.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:33 smithi196 ceph-mon[27428]: purged_snaps scrub ok 2024-09-17T12:19:33.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:33 smithi196 ceph-mon[27428]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:33.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:33 smithi196 ceph-mon[27428]: osd.5 [v2:172.21.15.196:6808/2204973118,v1:172.21.15.196:6809/2204973118] boot 2024-09-17T12:19:33.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:33 smithi196 ceph-mon[27428]: osdmap e39: 6 total, 6 up, 6 in 2024-09-17T12:19:33.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:33 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:33.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:33 smithi016 ceph-mon[23719]: purged_snaps scrub starts 2024-09-17T12:19:33.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:33 smithi016 ceph-mon[23719]: purged_snaps scrub ok 2024-09-17T12:19:33.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:33 smithi016 ceph-mon[23719]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:33.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:33 smithi016 ceph-mon[23719]: osd.5 [v2:172.21.15.196:6808/2204973118,v1:172.21.15.196:6809/2204973118] boot 2024-09-17T12:19:33.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:33 smithi016 ceph-mon[23719]: osdmap e39: 6 total, 6 up, 6 in 2024-09-17T12:19:33.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:33 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:33.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:33 smithi016 ceph-mon[34212]: purged_snaps scrub starts 2024-09-17T12:19:33.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:33 smithi016 ceph-mon[34212]: purged_snaps scrub ok 2024-09-17T12:19:33.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:33 smithi016 ceph-mon[34212]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T12:19:33.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:33 smithi016 ceph-mon[34212]: osd.5 [v2:172.21.15.196:6808/2204973118,v1:172.21.15.196:6809/2204973118] boot 2024-09-17T12:19:33.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:33 smithi016 ceph-mon[34212]: osdmap e39: 6 total, 6 up, 6 in 2024-09-17T12:19:33.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:33 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:19:34.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:34 smithi196 ceph-mon[27428]: osdmap e40: 6 total, 6 up, 6 in 2024-09-17T12:19:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:34 smithi196 ceph-mon[27428]: Detected new or changed devices on smithi196 2024-09-17T12:19:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:34 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:34 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:34 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:34 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:34 smithi196 ceph-mon[27428]: Adjusting osd_memory_target on smithi196 to 8544M 2024-09-17T12:19:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:34 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:34 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:34 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:34 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:34 smithi196 ceph-mon[27428]: osdmap e41: 6 total, 6 up, 6 in 2024-09-17T12:19:34.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[34212]: osdmap e40: 6 total, 6 up, 6 in 2024-09-17T12:19:34.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[34212]: Detected new or changed devices on smithi196 2024-09-17T12:19:34.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:34.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:34.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:34.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:34.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[34212]: Adjusting osd_memory_target on smithi196 to 8544M 2024-09-17T12:19:34.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:34.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:34.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:34.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:34.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[34212]: osdmap e41: 6 total, 6 up, 6 in 2024-09-17T12:19:34.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[23719]: osdmap e40: 6 total, 6 up, 6 in 2024-09-17T12:19:34.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[23719]: Detected new or changed devices on smithi196 2024-09-17T12:19:34.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:34.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:34.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:34.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:34.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[23719]: Adjusting osd_memory_target on smithi196 to 8544M 2024-09-17T12:19:34.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:34.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:34.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:34.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:34.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:34 smithi016 ceph-mon[23719]: osdmap e41: 6 total, 6 up, 6 in 2024-09-17T12:19:35.726 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-17T12:19:35.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_2 2024-09-17T12:19:35.937 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:19:36.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:35 smithi196 ceph-mon[27428]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:36.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:35 smithi196 ceph-mon[27428]: osdmap e42: 6 total, 6 up, 6 in 2024-09-17T12:19:36.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:35 smithi016 ceph-mon[23719]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:36.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:35 smithi016 ceph-mon[23719]: osdmap e42: 6 total, 6 up, 6 in 2024-09-17T12:19:36.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:35 smithi016 ceph-mon[34212]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:36.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:35 smithi016 ceph-mon[34212]: osdmap e42: 6 total, 6 up, 6 in 2024-09-17T12:19:37.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:37 smithi196 ceph-mon[27428]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:37.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:37 smithi196 ceph-mon[27428]: osdmap e43: 6 total, 6 up, 6 in 2024-09-17T12:19:37.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:37 smithi016 ceph-mon[23719]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:37.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:37 smithi016 ceph-mon[23719]: osdmap e43: 6 total, 6 up, 6 in 2024-09-17T12:19:37.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:37 smithi016 ceph-mon[34212]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:37.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:37 smithi016 ceph-mon[34212]: osdmap e43: 6 total, 6 up, 6 in 2024-09-17T12:19:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:38 smithi196 ceph-mon[27428]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:19:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:38 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:19:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:38 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:19:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:38 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:38.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:38 smithi016 ceph-mon[23719]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:19:38.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:38 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:19:38.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:38 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:19:38.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:38 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:38.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:38 smithi016 ceph-mon[34212]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:19:38.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:38 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:19:38.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:38 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:19:38.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:38 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:39.721 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:39 smithi196 ceph-mon[27428]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:39.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:39 smithi016 ceph-mon[23719]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:39.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:39 smithi016 ceph-mon[34212]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:40.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:40 smithi016 ceph-mon[23719]: from='client.? 172.21.15.196:0/1042320191' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7891e15-85f5-4729-9143-e34199f6c03e"}]: dispatch 2024-09-17T12:19:40.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:40 smithi016 ceph-mon[23719]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7891e15-85f5-4729-9143-e34199f6c03e"}]: dispatch 2024-09-17T12:19:40.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:40 smithi016 ceph-mon[23719]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7891e15-85f5-4729-9143-e34199f6c03e"}]': finished 2024-09-17T12:19:40.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:40 smithi016 ceph-mon[23719]: osdmap e44: 7 total, 6 up, 7 in 2024-09-17T12:19:40.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:40 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:40.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:40 smithi016 ceph-mon[34212]: from='client.? 172.21.15.196:0/1042320191' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7891e15-85f5-4729-9143-e34199f6c03e"}]: dispatch 2024-09-17T12:19:40.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:40 smithi016 ceph-mon[34212]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7891e15-85f5-4729-9143-e34199f6c03e"}]: dispatch 2024-09-17T12:19:40.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:40 smithi016 ceph-mon[34212]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7891e15-85f5-4729-9143-e34199f6c03e"}]': finished 2024-09-17T12:19:40.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:40 smithi016 ceph-mon[34212]: osdmap e44: 7 total, 6 up, 7 in 2024-09-17T12:19:40.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:40 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:41.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:40 smithi196 ceph-mon[27428]: from='client.? 172.21.15.196:0/1042320191' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7891e15-85f5-4729-9143-e34199f6c03e"}]: dispatch 2024-09-17T12:19:41.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:40 smithi196 ceph-mon[27428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7891e15-85f5-4729-9143-e34199f6c03e"}]: dispatch 2024-09-17T12:19:41.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:40 smithi196 ceph-mon[27428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7891e15-85f5-4729-9143-e34199f6c03e"}]': finished 2024-09-17T12:19:41.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:40 smithi196 ceph-mon[27428]: osdmap e44: 7 total, 6 up, 7 in 2024-09-17T12:19:41.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:40 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:41.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:41 smithi016 ceph-mon[23719]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:41.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:41 smithi016 ceph-mon[23719]: from='client.? 172.21.15.196:0/1120724844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:19:41.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:41 smithi016 ceph-mon[34212]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:41.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:41 smithi016 ceph-mon[34212]: from='client.? 172.21.15.196:0/1120724844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:19:42.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:41 smithi196 ceph-mon[27428]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:42.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:41 smithi196 ceph-mon[27428]: from='client.? 172.21.15.196:0/1120724844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:19:43.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:43 smithi196 ceph-mon[27428]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:43.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:43 smithi016 ceph-mon[23719]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:43.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:43 smithi016 ceph-mon[34212]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:45.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:45 smithi196 ceph-mon[27428]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:45.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:45 smithi016 ceph-mon[23719]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:45.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:45 smithi016 ceph-mon[34212]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:46.682 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:46 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T12:19:46.682 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:46 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:46.682 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:46 smithi196 ceph-mon[27428]: Deploying daemon osd.6 on smithi196 2024-09-17T12:19:46.842 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:46 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T12:19:46.843 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:46 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:46.843 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:46 smithi016 ceph-mon[23719]: Deploying daemon osd.6 on smithi196 2024-09-17T12:19:46.843 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:46 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T12:19:46.843 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:46 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:46.843 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:46 smithi016 ceph-mon[34212]: Deploying daemon osd.6 on smithi196 2024-09-17T12:19:47.771 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:47 smithi196 ceph-mon[27428]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:47.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:47 smithi016 ceph-mon[23719]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:47.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:47 smithi016 ceph-mon[34212]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:49.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:49 smithi196 ceph-mon[27428]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:49.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:49 smithi016 ceph-mon[23719]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:49.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:49 smithi016 ceph-mon[34212]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:51.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:51 smithi196 ceph-mon[27428]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:51.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:51 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:51.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:51 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:51.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:51 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:51.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:51 smithi016 ceph-mon[23719]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:51.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:51 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:51.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:51 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:51.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:51 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:51.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:51 smithi016 ceph-mon[34212]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:51.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:51 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:51.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:51 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:51.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:51 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:53.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:53 smithi196 ceph-mon[27428]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:53.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:53 smithi016 ceph-mon[23719]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:53.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:53 smithi016 ceph-mon[34212]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:54.719 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 6 on host 'smithi196' 2024-09-17T12:19:55.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:54 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:54 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:54 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:54 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:54 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:54 smithi196 ceph-mon[27428]: from='osd.6 [v2:172.21.15.196:6816/3128478817,v1:172.21.15.196:6817/3128478817]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T12:19:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:54 smithi196 ceph-mon[27428]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T12:19:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:54 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:54 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:54 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[23719]: from='osd.6 [v2:172.21.15.196:6816/3128478817,v1:172.21.15.196:6817/3128478817]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T12:19:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[23719]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T12:19:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[34212]: from='osd.6 [v2:172.21.15.196:6816/3128478817,v1:172.21.15.196:6817/3128478817]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T12:19:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[34212]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T12:19:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:19:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:54 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:55.294 DEBUG:teuthology.orchestra.run.smithi196:osd.6> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.6.service 2024-09-17T12:19:55.296 INFO:tasks.cephadm:Deploying osd.7 on smithi196 with /dev/vg_nvme/lv_1... 2024-09-17T12:19:55.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-17T12:19:56.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:55 smithi196 ceph-mon[27428]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:55 smithi196 ceph-mon[27428]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T12:19:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:55 smithi196 ceph-mon[27428]: osdmap e45: 7 total, 6 up, 7 in 2024-09-17T12:19:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:55 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:55 smithi196 ceph-mon[27428]: from='osd.6 [v2:172.21.15.196:6816/3128478817,v1:172.21.15.196:6817/3128478817]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:55 smithi196 ceph-mon[27428]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:56.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:55 smithi016 ceph-mon[34212]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:56.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:55 smithi016 ceph-mon[34212]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T12:19:56.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:55 smithi016 ceph-mon[34212]: osdmap e45: 7 total, 6 up, 7 in 2024-09-17T12:19:56.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:55 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:56.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:55 smithi016 ceph-mon[34212]: from='osd.6 [v2:172.21.15.196:6816/3128478817,v1:172.21.15.196:6817/3128478817]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:56.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:55 smithi016 ceph-mon[34212]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:55 smithi016 ceph-mon[23719]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:55 smithi016 ceph-mon[23719]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T12:19:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:55 smithi016 ceph-mon[23719]: osdmap e45: 7 total, 6 up, 7 in 2024-09-17T12:19:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:55 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:55 smithi016 ceph-mon[23719]: from='osd.6 [v2:172.21.15.196:6816/3128478817,v1:172.21.15.196:6817/3128478817]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:55 smithi016 ceph-mon[23719]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:19:56.669 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:19:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[43518]: 2024-09-17T12:19:56.370+0000 7f5b7d1a1640 -1 osd.6 0 waiting for initial osdmap 2024-09-17T12:19:56.670 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:19:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[43518]: 2024-09-17T12:19:56.376+0000 7f5b74fc2640 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:19:56.869 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:19:57.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:56 smithi196 ceph-mon[27428]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-17T12:19:57.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:56 smithi196 ceph-mon[27428]: osdmap e46: 7 total, 6 up, 7 in 2024-09-17T12:19:57.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:56 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:57.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:56 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:57.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:56 smithi196 ceph-mon[27428]: osdmap e47: 7 total, 6 up, 7 in 2024-09-17T12:19:57.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:56 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:57.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:56 smithi196 ceph-mon[27428]: from='osd.6 ' entity='osd.6' 2024-09-17T12:19:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[23719]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-17T12:19:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[23719]: osdmap e46: 7 total, 6 up, 7 in 2024-09-17T12:19:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:57.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:57.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[23719]: osdmap e47: 7 total, 6 up, 7 in 2024-09-17T12:19:57.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:57.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[23719]: from='osd.6 ' entity='osd.6' 2024-09-17T12:19:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[34212]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-17T12:19:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[34212]: osdmap e46: 7 total, 6 up, 7 in 2024-09-17T12:19:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[34212]: osdmap e47: 7 total, 6 up, 7 in 2024-09-17T12:19:57.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:57.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:56 smithi016 ceph-mon[34212]: from='osd.6 ' entity='osd.6' 2024-09-17T12:19:58.180 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:57 smithi196 ceph-mon[27428]: purged_snaps scrub starts 2024-09-17T12:19:58.180 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:57 smithi196 ceph-mon[27428]: purged_snaps scrub ok 2024-09-17T12:19:58.180 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:57 smithi196 ceph-mon[27428]: pgmap v148: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:58.180 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:57 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:58.180 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:57 smithi196 ceph-mon[27428]: osd.6 [v2:172.21.15.196:6816/3128478817,v1:172.21.15.196:6817/3128478817] boot 2024-09-17T12:19:58.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:57 smithi196 ceph-mon[27428]: osdmap e48: 7 total, 7 up, 7 in 2024-09-17T12:19:58.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:57 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:58.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[23719]: purged_snaps scrub starts 2024-09-17T12:19:58.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[23719]: purged_snaps scrub ok 2024-09-17T12:19:58.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[23719]: pgmap v148: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:58.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:58.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[23719]: osd.6 [v2:172.21.15.196:6816/3128478817,v1:172.21.15.196:6817/3128478817] boot 2024-09-17T12:19:58.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[23719]: osdmap e48: 7 total, 7 up, 7 in 2024-09-17T12:19:58.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:58.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[34212]: purged_snaps scrub starts 2024-09-17T12:19:58.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[34212]: purged_snaps scrub ok 2024-09-17T12:19:58.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[34212]: pgmap v148: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T12:19:58.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:58.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[34212]: osd.6 [v2:172.21.15.196:6816/3128478817,v1:172.21.15.196:6817/3128478817] boot 2024-09-17T12:19:58.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[34212]: osdmap e48: 7 total, 7 up, 7 in 2024-09-17T12:19:58.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:57 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:19:59.233 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:59 smithi196 ceph-mon[27428]: osdmap e49: 7 total, 7 up, 7 in 2024-09-17T12:19:59.233 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:59 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:59.233 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:59 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:59.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:59 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:59.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:59 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:59.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:59 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:59.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:59 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:59.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:59 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:59.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:59 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:59.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:19:59 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:59.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[23719]: osdmap e49: 7 total, 7 up, 7 in 2024-09-17T12:19:59.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:59.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:59.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:59.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:59.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:59.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:59.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:59.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:59.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:59.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[34212]: osdmap e49: 7 total, 7 up, 7 in 2024-09-17T12:19:59.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:59.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:59.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:59.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:59.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:19:59.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:59.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:19:59.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:19:59.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:19:59 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:19:59.951 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-17T12:19:59.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_1 2024-09-17T12:20:00.171 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:20:00.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:00 smithi196 ceph-mon[27428]: pgmap v151: 1 pgs: 1 unknown; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:00.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:00 smithi196 ceph-mon[27428]: Detected new or changed devices on smithi196 2024-09-17T12:20:00.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:00 smithi196 ceph-mon[27428]: Adjusting osd_memory_target on smithi196 to 5696M 2024-09-17T12:20:00.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:00 smithi196 ceph-mon[27428]: osdmap e50: 7 total, 7 up, 7 in 2024-09-17T12:20:00.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:00 smithi196 ceph-mon[27428]: overall HEALTH_OK 2024-09-17T12:20:00.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:00 smithi016 ceph-mon[34212]: pgmap v151: 1 pgs: 1 unknown; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:00.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:00 smithi016 ceph-mon[34212]: Detected new or changed devices on smithi196 2024-09-17T12:20:00.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:00 smithi016 ceph-mon[34212]: Adjusting osd_memory_target on smithi196 to 5696M 2024-09-17T12:20:00.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:00 smithi016 ceph-mon[34212]: osdmap e50: 7 total, 7 up, 7 in 2024-09-17T12:20:00.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:00 smithi016 ceph-mon[34212]: overall HEALTH_OK 2024-09-17T12:20:00.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:00 smithi016 ceph-mon[23719]: pgmap v151: 1 pgs: 1 unknown; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:00.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:00 smithi016 ceph-mon[23719]: Detected new or changed devices on smithi196 2024-09-17T12:20:00.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:00 smithi016 ceph-mon[23719]: Adjusting osd_memory_target on smithi196 to 5696M 2024-09-17T12:20:00.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:00 smithi016 ceph-mon[23719]: osdmap e50: 7 total, 7 up, 7 in 2024-09-17T12:20:00.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:00 smithi016 ceph-mon[23719]: overall HEALTH_OK 2024-09-17T12:20:01.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:01 smithi196 ceph-mon[27428]: osdmap e51: 7 total, 7 up, 7 in 2024-09-17T12:20:01.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:01 smithi016 ceph-mon[23719]: osdmap e51: 7 total, 7 up, 7 in 2024-09-17T12:20:01.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:01 smithi016 ceph-mon[34212]: osdmap e51: 7 total, 7 up, 7 in 2024-09-17T12:20:02.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:02 smithi196 ceph-mon[27428]: pgmap v154: 1 pgs: 1 unknown; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:02.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:02 smithi196 ceph-mon[27428]: osdmap e52: 7 total, 7 up, 7 in 2024-09-17T12:20:02.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:20:02.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:20:02.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:02 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:02.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:02 smithi016 ceph-mon[23719]: pgmap v154: 1 pgs: 1 unknown; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:02.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:02 smithi016 ceph-mon[23719]: osdmap e52: 7 total, 7 up, 7 in 2024-09-17T12:20:02.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:20:02.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:20:02.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:02 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:02.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:02 smithi016 ceph-mon[34212]: pgmap v154: 1 pgs: 1 unknown; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:02.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:02 smithi016 ceph-mon[34212]: osdmap e52: 7 total, 7 up, 7 in 2024-09-17T12:20:02.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T12:20:02.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T12:20:02.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:02 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:03.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:03 smithi196 ceph-mon[27428]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:03 smithi016 ceph-mon[23719]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:03.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:03 smithi016 ceph-mon[34212]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:04.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:04 smithi016 ceph-mon[23719]: pgmap v156: 1 pgs: 1 unknown; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:04.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:04 smithi016 ceph-mon[34212]: pgmap v156: 1 pgs: 1 unknown; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:04.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:04 smithi196 ceph-mon[27428]: pgmap v156: 1 pgs: 1 unknown; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:05.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:05 smithi196 ceph-mon[27428]: from='client.? 172.21.15.196:0/3964330256' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5698cae8-7738-4933-a108-4b039e629641"}]: dispatch 2024-09-17T12:20:05.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:05 smithi196 ceph-mon[27428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5698cae8-7738-4933-a108-4b039e629641"}]: dispatch 2024-09-17T12:20:05.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:05 smithi196 ceph-mon[27428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5698cae8-7738-4933-a108-4b039e629641"}]': finished 2024-09-17T12:20:05.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:05 smithi196 ceph-mon[27428]: osdmap e53: 8 total, 7 up, 8 in 2024-09-17T12:20:05.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:05 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:05.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:05 smithi196 ceph-mon[27428]: from='client.? 172.21.15.196:0/1397598290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:20:05.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:05 smithi016 ceph-mon[23719]: from='client.? 172.21.15.196:0/3964330256' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5698cae8-7738-4933-a108-4b039e629641"}]: dispatch 2024-09-17T12:20:05.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:05 smithi016 ceph-mon[23719]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5698cae8-7738-4933-a108-4b039e629641"}]: dispatch 2024-09-17T12:20:05.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:05 smithi016 ceph-mon[23719]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5698cae8-7738-4933-a108-4b039e629641"}]': finished 2024-09-17T12:20:05.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:05 smithi016 ceph-mon[23719]: osdmap e53: 8 total, 7 up, 8 in 2024-09-17T12:20:05.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:05 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:05.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:05 smithi016 ceph-mon[23719]: from='client.? 172.21.15.196:0/1397598290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:20:05.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:05 smithi016 ceph-mon[34212]: from='client.? 172.21.15.196:0/3964330256' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5698cae8-7738-4933-a108-4b039e629641"}]: dispatch 2024-09-17T12:20:05.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:05 smithi016 ceph-mon[34212]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5698cae8-7738-4933-a108-4b039e629641"}]: dispatch 2024-09-17T12:20:05.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:05 smithi016 ceph-mon[34212]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5698cae8-7738-4933-a108-4b039e629641"}]': finished 2024-09-17T12:20:05.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:05 smithi016 ceph-mon[34212]: osdmap e53: 8 total, 7 up, 8 in 2024-09-17T12:20:05.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:05 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:05.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:05 smithi016 ceph-mon[34212]: from='client.? 172.21.15.196:0/1397598290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T12:20:06.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:06 smithi016 ceph-mon[23719]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-09-17T12:20:06.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:06 smithi016 ceph-mon[34212]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-09-17T12:20:06.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:06 smithi196 ceph-mon[27428]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-09-17T12:20:08.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:08 smithi196 ceph-mon[27428]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-17T12:20:08.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:08 smithi016 ceph-mon[23719]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-17T12:20:08.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:08 smithi016 ceph-mon[34212]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-17T12:20:10.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:10 smithi196 ceph-mon[27428]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-17T12:20:10.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:10 smithi016 ceph-mon[23719]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-17T12:20:10.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:10 smithi016 ceph-mon[34212]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-17T12:20:11.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:11 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T12:20:11.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:11 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:11.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:11 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T12:20:11.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:11 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:11.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:11 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T12:20:11.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:11 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:12.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:12 smithi016 ceph-mon[23719]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-17T12:20:12.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:12 smithi016 ceph-mon[23719]: Deploying daemon osd.7 on smithi196 2024-09-17T12:20:12.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:12 smithi016 ceph-mon[34212]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-17T12:20:12.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:12 smithi016 ceph-mon[34212]: Deploying daemon osd.7 on smithi196 2024-09-17T12:20:12.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:12 smithi196 ceph-mon[27428]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-17T12:20:12.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:12 smithi196 ceph-mon[27428]: Deploying daemon osd.7 on smithi196 2024-09-17T12:20:14.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:14 smithi016 ceph-mon[23719]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T12:20:14.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:14 smithi016 ceph-mon[34212]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T12:20:14.715 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:14 smithi196 ceph-mon[27428]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T12:20:16.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:16 smithi196 ceph-mon[27428]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:16.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:16 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:20:16.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:16 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:16.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:16 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:16.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:16 smithi016 ceph-mon[23719]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:16.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:16 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:20:16.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:16 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:16.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:16 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:16.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:16 smithi016 ceph-mon[34212]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:16.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:16 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:20:16.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:16 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:16.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:16 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:18.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:18 smithi196 ceph-mon[27428]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:18.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:18 smithi016 ceph-mon[23719]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:18.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:18 smithi016 ceph-mon[34212]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:18.973 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 7 on host 'smithi196' 2024-09-17T12:20:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:19 smithi196 ceph-mon[27428]: from='osd.7 [v2:172.21.15.196:6824/827333513,v1:172.21.15.196:6825/827333513]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T12:20:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:19 smithi196 ceph-mon[27428]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T12:20:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:19 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:19 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:19.662 DEBUG:teuthology.orchestra.run.smithi196:osd.7> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.7.service 2024-09-17T12:20:19.664 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-17T12:20:19.664 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-17T12:20:19.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:19 smithi016 ceph-mon[23719]: from='osd.7 [v2:172.21.15.196:6824/827333513,v1:172.21.15.196:6825/827333513]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T12:20:19.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:19 smithi016 ceph-mon[23719]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T12:20:19.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:19 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:19.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:19 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:19.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:19 smithi016 ceph-mon[34212]: from='osd.7 [v2:172.21.15.196:6824/827333513,v1:172.21.15.196:6825/827333513]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T12:20:19.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:19 smithi016 ceph-mon[34212]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T12:20:19.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:19 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:19.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:19 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:19.887 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:20:20.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:20 smithi196 ceph-mon[27428]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:20.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:20 smithi196 ceph-mon[27428]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T12:20:20.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:20 smithi196 ceph-mon[27428]: osdmap e54: 8 total, 7 up, 8 in 2024-09-17T12:20:20.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:20 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:20.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:20 smithi196 ceph-mon[27428]: from='osd.7 [v2:172.21.15.196:6824/827333513,v1:172.21.15.196:6825/827333513]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:20:20.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:20 smithi196 ceph-mon[27428]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:20:20.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:20 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:20.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:20 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:20.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[23719]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:20.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[23719]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T12:20:20.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[23719]: osdmap e54: 8 total, 7 up, 8 in 2024-09-17T12:20:20.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:20.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[23719]: from='osd.7 [v2:172.21.15.196:6824/827333513,v1:172.21.15.196:6825/827333513]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:20:20.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[23719]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:20:20.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:20.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:20.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[34212]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:20.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[34212]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T12:20:20.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[34212]: osdmap e54: 8 total, 7 up, 8 in 2024-09-17T12:20:20.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:20.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[34212]: from='osd.7 [v2:172.21.15.196:6824/827333513,v1:172.21.15.196:6825/827333513]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:20:20.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[34212]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-17T12:20:20.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:20.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:20 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:21.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:20:20 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[49245]: 2024-09-17T12:20:20.833+0000 7f2055552640 -1 osd.7 0 waiting for initial osdmap 2024-09-17T12:20:21.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:20:20 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[49245]: 2024-09-17T12:20:20.839+0000 7f204d373640 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:20:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[34212]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-17T12:20:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[34212]: osdmap e55: 8 total, 7 up, 8 in 2024-09-17T12:20:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[34212]: from='osd.7 ' entity='osd.7' 2024-09-17T12:20:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[34212]: osd.7 [v2:172.21.15.196:6824/827333513,v1:172.21.15.196:6825/827333513] boot 2024-09-17T12:20:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[34212]: osdmap e56: 8 total, 8 up, 8 in 2024-09-17T12:20:21.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[23719]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-17T12:20:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[23719]: osdmap e55: 8 total, 7 up, 8 in 2024-09-17T12:20:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[23719]: from='osd.7 ' entity='osd.7' 2024-09-17T12:20:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[23719]: osd.7 [v2:172.21.15.196:6824/827333513,v1:172.21.15.196:6825/827333513] boot 2024-09-17T12:20:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[23719]: osdmap e56: 8 total, 8 up, 8 in 2024-09-17T12:20:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:21 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:21.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:21 smithi196 ceph-mon[27428]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-17T12:20:21.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:21 smithi196 ceph-mon[27428]: osdmap e55: 8 total, 7 up, 8 in 2024-09-17T12:20:21.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:21 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:21.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:21 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:21.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:21 smithi196 ceph-mon[27428]: from='osd.7 ' entity='osd.7' 2024-09-17T12:20:21.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:21 smithi196 ceph-mon[27428]: osd.7 [v2:172.21.15.196:6824/827333513,v1:172.21.15.196:6825/827333513] boot 2024-09-17T12:20:21.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:21 smithi196 ceph-mon[27428]: osdmap e56: 8 total, 8 up, 8 in 2024-09-17T12:20:21.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:21 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:21.959 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:20:22.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:22 smithi016 ceph-mon[23719]: purged_snaps scrub starts 2024-09-17T12:20:22.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:22 smithi016 ceph-mon[23719]: purged_snaps scrub ok 2024-09-17T12:20:22.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:22 smithi016 ceph-mon[23719]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:22.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:22 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1059205592' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T12:20:22.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:22 smithi016 ceph-mon[23719]: osdmap e57: 8 total, 8 up, 8 in 2024-09-17T12:20:22.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:22 smithi016 ceph-mon[34212]: purged_snaps scrub starts 2024-09-17T12:20:22.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:22 smithi016 ceph-mon[34212]: purged_snaps scrub ok 2024-09-17T12:20:22.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:22 smithi016 ceph-mon[34212]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:22.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:22 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1059205592' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T12:20:22.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:22 smithi016 ceph-mon[34212]: osdmap e57: 8 total, 8 up, 8 in 2024-09-17T12:20:22.766 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1726575621,"num_in_osds":8,"osd_in_since":1726575604,"num_remapped_pgs":1} 2024-09-17T12:20:22.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T12:20:22.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:22 smithi196 ceph-mon[27428]: purged_snaps scrub starts 2024-09-17T12:20:22.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:22 smithi196 ceph-mon[27428]: purged_snaps scrub ok 2024-09-17T12:20:22.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:22 smithi196 ceph-mon[27428]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T12:20:22.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:22 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1059205592' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T12:20:22.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:22 smithi196 ceph-mon[27428]: osdmap e57: 8 total, 8 up, 8 in 2024-09-17T12:20:23.015 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:20:24.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:24.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: osdmap e58: 8 total, 8 up, 8 in 2024-09-17T12:20:24.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: Detected new or changed devices on smithi196 2024-09-17T12:20:24.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:24.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:24.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:24.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:24.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: Adjusting osd_memory_target on smithi196 to 4272M 2024-09-17T12:20:24.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:24.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:20:24.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:20:24.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:24.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:20:24.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:24.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: osdmap e58: 8 total, 8 up, 8 in 2024-09-17T12:20:24.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: Detected new or changed devices on smithi196 2024-09-17T12:20:24.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:24.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:24.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:24.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:24.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: Adjusting osd_memory_target on smithi196 to 4272M 2024-09-17T12:20:24.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:24.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:20:24.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:20:24.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:24.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:20:24.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:24 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: osdmap e58: 8 total, 8 up, 8 in 2024-09-17T12:20:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: Detected new or changed devices on smithi196 2024-09-17T12:20:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: Adjusting osd_memory_target on smithi196 to 4272M 2024-09-17T12:20:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:24.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:20:24.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:20:24.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:24.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:20:24.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:24 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:20:24.990 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":59,"fsid":"5b9f4610-74ee-11ef-bceb-c7b262605968","created":"2024-09-17T12:15:25.688415+0000","modified":"2024-09-17T12:20:24.279999+0000","last_up_change":"2024-09-17T12:20:21.089433+0000","last_in_change":"2024-09-17T12:20:04.339833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T12:18:24.896429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32f1f69d-aaf7-42ed-92f0-9d1aa8bdd3c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":869442370},{"type":"v1","addr":"172.21.15.16:6803","nonce":869442370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":869442370},{"type":"v1","addr":"172.21.15.16:6805","nonce":869442370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":869442370},{"type":"v1","addr":"172.21.15.16:6809","nonce":869442370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":869442370},{"type":"v1","addr":"172.21.15.16:6807","nonce":869442370}]},"public_addr":"172.21.15.16:6803/869442370","cluster_addr":"172.21.15.16:6805/869442370","heartbeat_back_addr":"172.21.15.16:6809/869442370","heartbeat_front_addr":"172.21.15.16:6807/869442370","state":["exists","up"]},{"osd":1,"uuid":"b2dc9d4d-bcc8-4492-af56-57caea2ec732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":755127111},{"type":"v1","addr":"172.21.15.16:6811","nonce":755127111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":755127111},{"type":"v1","addr":"172.21.15.16:6813","nonce":755127111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":755127111},{"type":"v1","addr":"172.21.15.16:6817","nonce":755127111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":755127111},{"type":"v1","addr":"172.21.15.16:6815","nonce":755127111}]},"public_addr":"172.21.15.16:6811/755127111","cluster_addr":"172.21.15.16:6813/755127111","heartbeat_back_addr":"172.21.15.16:6817/755127111","heartbeat_front_addr":"172.21.15.16:6815/755127111","state":["exists","up"]},{"osd":2,"uuid":"aa7dc4a4-7b6f-440c-bdd8-78f7f1786434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2313749493},{"type":"v1","addr":"172.21.15.16:6819","nonce":2313749493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2313749493},{"type":"v1","addr":"172.21.15.16:6821","nonce":2313749493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2313749493},{"type":"v1","addr":"172.21.15.16:6825","nonce":2313749493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2313749493},{"type":"v1","addr":"172.21.15.16:6823","nonce":2313749493}]},"public_addr":"172.21.15.16:6819/2313749493","cluster_addr":"172.21.15.16:6821/2313749493","heartbeat_back_addr":"172.21.15.16:6825/2313749493","heartbeat_front_addr":"172.21.15.16:6823/2313749493","state":["exists","up"]},{"osd":3,"uuid":"81f6d250-32a6-4276-b6bc-aac7e27e1a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3579869089},{"type":"v1","addr":"172.21.15.16:6827","nonce":3579869089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3579869089},{"type":"v1","addr":"172.21.15.16:6829","nonce":3579869089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3579869089},{"type":"v1","addr":"172.21.15.16:6833","nonce":3579869089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3579869089},{"type":"v1","addr":"172.21.15.16:6831","nonce":3579869089}]},"public_addr":"172.21.15.16:6827/3579869089","cluster_addr":"172.21.15.16:6829/3579869089","heartbeat_back_addr":"172.21.15.16:6833/3579869089","heartbeat_front_addr":"172.21.15.16:6831/3579869089","state":["exists","up"]},{"osd":4,"uuid":"e6198e3b-3a1d-40a0-b3e8-9d34fc42ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":590253655},{"type":"v1","addr":"172.21.15.196:6801","nonce":590253655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":590253655},{"type":"v1","addr":"172.21.15.196:6803","nonce":590253655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":590253655},{"type":"v1","addr":"172.21.15.196:6807","nonce":590253655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":590253655},{"type":"v1","addr":"172.21.15.196:6805","nonce":590253655}]},"public_addr":"172.21.15.196:6801/590253655","cluster_addr":"172.21.15.196:6803/590253655","heartbeat_back_addr":"172.21.15.196:6807/590253655","heartbeat_front_addr":"172.21.15.196:6805/590253655","state":["exists","up"]},{"osd":5,"uuid":"7eaa3677-35e9-4ce5-8515-ca7a27286824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2204973118},{"type":"v1","addr":"172.21.15.196:6809","nonce":2204973118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2204973118},{"type":"v1","addr":"172.21.15.196:6811","nonce":2204973118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2204973118},{"type":"v1","addr":"172.21.15.196:6815","nonce":2204973118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2204973118},{"type":"v1","addr":"172.21.15.196:6813","nonce":2204973118}]},"public_addr":"172.21.15.196:6809/2204973118","cluster_addr":"172.21.15.196:6811/2204973118","heartbeat_back_addr":"172.21.15.196:6815/2204973118","heartbeat_front_addr":"172.21.15.196:6813/2204973118","state":["exists","up"]},{"osd":6,"uuid":"f7891e15-85f5-4729-9143-e34199f6c03e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3128478817},{"type":"v1","addr":"172.21.15.196:6817","nonce":3128478817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3128478817},{"type":"v1","addr":"172.21.15.196:6819","nonce":3128478817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3128478817},{"type":"v1","addr":"172.21.15.196:6823","nonce":3128478817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3128478817},{"type":"v1","addr":"172.21.15.196:6821","nonce":3128478817}]},"public_addr":"172.21.15.196:6817/3128478817","cluster_addr":"172.21.15.196:6819/3128478817","heartbeat_back_addr":"172.21.15.196:6823/3128478817","heartbeat_front_addr":"172.21.15.196:6821/3128478817","state":["exists","up"]},{"osd":7,"uuid":"5698cae8-7738-4933-a108-4b039e629641","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":827333513},{"type":"v1","addr":"172.21.15.196:6825","nonce":827333513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":827333513},{"type":"v1","addr":"172.21.15.196:6827","nonce":827333513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":827333513},{"type":"v1","addr":"172.21.15.196:6831","nonce":827333513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":827333513},{"type":"v1","addr":"172.21.15.196:6829","nonce":827333513}]},"public_addr":"172.21.15.196:6825/827333513","cluster_addr":"172.21.15.196:6827/827333513","heartbeat_back_addr":"172.21.15.196:6831/827333513","heartbeat_front_addr":"172.21.15.196:6829/827333513","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:17:33.036896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:17:56.357381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:18:20.441837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:18:44.476130+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:19:06.358889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:19:30.976024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:19:55.394849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:20:19.670476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/218077182":"2024-09-18T12:16:14.402306+0000","172.21.15.16:6801/4014507619":"2024-09-18T12:16:14.402306+0000","172.21.15.16:6800/4014507619":"2024-09-18T12:16:14.402306+0000","172.21.15.16:0/29394240":"2024-09-18T12:16:14.402306+0000","172.21.15.16:6801/3212434883":"2024-09-18T12:15:52.139948+0000","172.21.15.16:6800/3212434883":"2024-09-18T12:15:52.139948+0000","172.21.15.16:0/610303924":"2024-09-18T12:15:52.139948+0000","172.21.15.16:0/3411803517":"2024-09-18T12:16:14.402306+0000","172.21.15.16:0/843792520":"2024-09-18T12:15:52.139948+0000","172.21.15.16:0/1310624606":"2024-09-18T12:15:52.139948+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T12:20:25.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:25 smithi016 ceph-mon[23719]: osdmap e59: 8 total, 8 up, 8 in 2024-09-17T12:20:25.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:25 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2357703797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T12:20:25.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:25 smithi016 ceph-mon[34212]: osdmap e59: 8 total, 8 up, 8 in 2024-09-17T12:20:25.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:25 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2357703797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T12:20:25.611 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T12:18:24.896429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T12:20:25.611 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-17T12:20:25.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:25 smithi196 ceph-mon[27428]: osdmap e59: 8 total, 8 up, 8 in 2024-09-17T12:20:25.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:25 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2357703797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T12:20:25.815 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:20:26.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:26 smithi016 ceph-mon[23719]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:26.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:26 smithi016 ceph-mon[23719]: osdmap e60: 8 total, 8 up, 8 in 2024-09-17T12:20:26.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:26 smithi016 ceph-mon[34212]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:26.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:26 smithi016 ceph-mon[34212]: osdmap e60: 8 total, 8 up, 8 in 2024-09-17T12:20:26.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:26 smithi196 ceph-mon[27428]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:26.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:26 smithi196 ceph-mon[27428]: osdmap e60: 8 total, 8 up, 8 in 2024-09-17T12:20:27.587 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-09-17T12:20:28.244 INFO:tasks.cephadm:Adding prometheus.a on smithi196 2024-09-17T12:20:28.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi196=a' 2024-09-17T12:20:28.454 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:20:28.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:28 smithi196 ceph-mon[27428]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-09-17T12:20:28.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:28 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3621500759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-17T12:20:28.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:28 smithi016 ceph-mon[34212]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-09-17T12:20:28.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:28 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3621500759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-17T12:20:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:28 smithi016 ceph-mon[23719]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-09-17T12:20:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:28 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3621500759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-17T12:20:30.179 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled prometheus update... 2024-09-17T12:20:30.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:30 smithi196 ceph-mon[27428]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T12:20:30.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:30 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:30.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:30 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:20:30.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:30 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:30.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:30 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:20:30.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:30 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:30.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:30 smithi016 ceph-mon[23719]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T12:20:30.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:30.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:20:30.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:30.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:20:30.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:30 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:30.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:30 smithi016 ceph-mon[34212]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T12:20:30.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:30 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:30.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:30 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:20:30.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:30 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:30.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:30 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:20:30.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:30 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:30.815 DEBUG:teuthology.orchestra.run.smithi196:prometheus.a> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-17T12:20:30.817 INFO:tasks.cephadm:Adding node-exporter.a on smithi016 2024-09-17T12:20:30.817 INFO:tasks.cephadm:Adding node-exporter.b on smithi196 2024-09-17T12:20:30.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi016=a;smithi196=b' 2024-09-17T12:20:31.059 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:20:31.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:31 smithi196 ceph-mon[27428]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:31.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:31 smithi196 ceph-mon[27428]: Saving service prometheus spec with placement smithi196=a;count:1 2024-09-17T12:20:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:31 smithi016 ceph-mon[23719]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:31 smithi016 ceph-mon[23719]: Saving service prometheus spec with placement smithi196=a;count:1 2024-09-17T12:20:31.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:31 smithi016 ceph-mon[34212]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:31.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:31 smithi016 ceph-mon[34212]: Saving service prometheus spec with placement smithi196=a;count:1 2024-09-17T12:20:32.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:32 smithi196 ceph-mon[27428]: Deploying daemon prometheus.a on smithi196 2024-09-17T12:20:32.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:32 smithi196 ceph-mon[27428]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-17T12:20:32.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:32 smithi016 ceph-mon[23719]: Deploying daemon prometheus.a on smithi196 2024-09-17T12:20:32.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:32 smithi016 ceph-mon[23719]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-17T12:20:32.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:32 smithi016 ceph-mon[34212]: Deploying daemon prometheus.a on smithi196 2024-09-17T12:20:32.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:32 smithi016 ceph-mon[34212]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-17T12:20:33.471 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled node-exporter update... 2024-09-17T12:20:34.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:34 smithi016 ceph-mon[23719]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-17T12:20:34.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:34 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:34.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:34 smithi016 ceph-mon[34212]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-17T12:20:34.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:34 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:34.497 DEBUG:teuthology.orchestra.run.smithi016:node-exporter.a> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-17T12:20:34.500 DEBUG:teuthology.orchestra.run.smithi196:node-exporter.b> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-17T12:20:34.503 INFO:tasks.cephadm:Adding alertmanager.a on smithi016 2024-09-17T12:20:34.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi016=a' 2024-09-17T12:20:34.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:34 smithi196 ceph-mon[27428]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-17T12:20:34.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:34 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:34.740 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:20:35.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:35 smithi196 ceph-mon[27428]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi016=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:35.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:35 smithi196 ceph-mon[27428]: Saving service node-exporter spec with placement smithi016=a;smithi196=b;count:2 2024-09-17T12:20:35.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:35 smithi016 ceph-mon[23719]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi016=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:35.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:35 smithi016 ceph-mon[23719]: Saving service node-exporter spec with placement smithi016=a;smithi196=b;count:2 2024-09-17T12:20:35.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:35 smithi016 ceph-mon[34212]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi016=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:35.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:35 smithi016 ceph-mon[34212]: Saving service node-exporter spec with placement smithi016=a;smithi196=b;count:2 2024-09-17T12:20:36.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:36 smithi196 ceph-mon[27428]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T12:20:36.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:36 smithi016 ceph-mon[23719]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T12:20:36.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:36 smithi016 ceph-mon[34212]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T12:20:37.334 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled alertmanager update... 2024-09-17T12:20:38.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:38 smithi196 ceph-mon[27428]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-17T12:20:38.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:38 smithi196 ceph-mon[27428]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi016=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:38.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:38 smithi196 ceph-mon[27428]: Saving service alertmanager spec with placement smithi016=a;count:1 2024-09-17T12:20:38.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:38 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:38.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:38 smithi016 ceph-mon[23719]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-17T12:20:38.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:38 smithi016 ceph-mon[23719]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi016=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:38.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:38 smithi016 ceph-mon[23719]: Saving service alertmanager spec with placement smithi016=a;count:1 2024-09-17T12:20:38.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:38 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:38 smithi016 ceph-mon[34212]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-17T12:20:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:38 smithi016 ceph-mon[34212]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi016=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:38 smithi016 ceph-mon[34212]: Saving service alertmanager spec with placement smithi016=a;count:1 2024-09-17T12:20:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:38 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:38.793 DEBUG:teuthology.orchestra.run.smithi016:alertmanager.a> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-17T12:20:38.797 INFO:tasks.cephadm:Adding grafana.a on smithi196 2024-09-17T12:20:38.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi196=a' 2024-09-17T12:20:39.373 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:20:40.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:40 smithi016 ceph-mon[34212]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:40.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:40 smithi016 ceph-mon[23719]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:40.714 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:40 smithi196 ceph-mon[27428]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:42.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:42 smithi196 ceph-mon[27428]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:42.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:42 smithi016 ceph-mon[23719]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:42.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:42 smithi016 ceph-mon[34212]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:43.639 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled grafana update... 2024-09-17T12:20:44.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:44 smithi196 ceph-mon[27428]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:44.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:44 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:44.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:44 smithi016 ceph-mon[23719]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:44.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:44 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:44.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:44 smithi016 ceph-mon[34212]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:44.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:44 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:44.867 DEBUG:teuthology.orchestra.run.smithi196:grafana.a> sudo journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@grafana.a.service 2024-09-17T12:20:44.869 INFO:tasks.cephadm:Setting up client nodes... 2024-09-17T12:20:44.869 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T12:20:45.082 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:20:45.374 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:45 smithi196 ceph-mon[27428]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:45.374 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:45 smithi196 ceph-mon[27428]: Saving service grafana spec with placement smithi196=a;count:1 2024-09-17T12:20:45.480 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:45 smithi016 ceph-mon[23719]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:45.480 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:45 smithi016 ceph-mon[23719]: Saving service grafana spec with placement smithi196=a;count:1 2024-09-17T12:20:45.480 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:45 smithi016 ceph-mon[34212]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:20:45.480 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:45 smithi016 ceph-mon[34212]: Saving service grafana spec with placement smithi196=a;count:1 2024-09-17T12:20:45.776 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:20:45 smithi196 systemd[1]: Started Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:20:46.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-mon[23719]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:46.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:46.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:46.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:46.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-17T12:20:46.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-mon[34212]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:46.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:46.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:46.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:46.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-17T12:20:46.678 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: ignoring --setuser ceph since I am not root 2024-09-17T12:20:46.679 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: ignoring --setgroup ceph since I am not root 2024-09-17T12:20:46.679 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:46 smithi196 ceph-mgr[29270]: -- 172.21.15.196:0/2083001146 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563c90f92d00 con 0x563c90f67400 2024-09-17T12:20:46.679 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:46 smithi196 ceph-mon[27428]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:46.679 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:46 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:46.679 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:46 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:46.679 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:46 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' 2024-09-17T12:20:46.679 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:46 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-17T12:20:46.748 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ignoring --setuser ceph since I am not root 2024-09-17T12:20:46.749 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ignoring --setgroup ceph since I am not root 2024-09-17T12:20:46.749 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:46.686+0000 7fbf71a13040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T12:20:46.749 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:46.747+0000 7fbf71a13040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T12:20:46.927 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-09-17T12:20:46.927 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAedOlmlRX4NhAAuFziJSUZ1Q0glBG2IlnQMQ== 2024-09-17T12:20:46.949 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:46.677+0000 7f42ba0d5040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T12:20:46.949 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:46.734+0000 7f42ba0d5040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T12:20:46.949 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:46.882+0000 7f42ba0d5040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T12:20:46.950 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:46.948+0000 7f42ba0d5040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T12:20:47.138 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:46.904+0000 7fbf71a13040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T12:20:47.138 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:46.984+0000 7fbf71a13040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T12:20:47.232 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:47 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:47.092+0000 7f42ba0d5040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:20:47.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:47.137+0000 7fbf71a13040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:20:47.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:47.286+0000 7fbf71a13040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T12:20:47.526 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:47 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:47.230+0000 7f42ba0d5040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T12:20:47.657 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-17T12:20:47.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-17T12:20:47.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T12:20:47.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T12:20:47.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-mon[23719]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-17T12:20:47.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-mon[23719]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-17T12:20:47.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/788438307' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T12:20:47.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-mon[23719]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T12:20:47.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-mon[23719]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T12:20:47.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-mon[34212]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-17T12:20:47.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-mon[34212]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-17T12:20:47.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/788438307' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T12:20:47.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-mon[34212]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T12:20:47.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-mon[34212]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T12:20:47.884 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:47.615+0000 7fbf71a13040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T12:20:47.885 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:47.684+0000 7fbf71a13040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T12:20:47.885 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:47.882+0000 7fbf71a13040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T12:20:47.905 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T12:20:48.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:47 smithi196 ceph-mon[27428]: from='mgr.14148 172.21.15.16:0/2506969895' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-17T12:20:48.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:47 smithi196 ceph-mon[27428]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-17T12:20:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:47 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/788438307' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T12:20:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:47 smithi196 ceph-mon[27428]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T12:20:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:47 smithi196 ceph-mon[27428]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T12:20:48.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:47 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:47.533+0000 7f42ba0d5040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T12:20:48.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:47 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:47.593+0000 7f42ba0d5040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T12:20:48.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:47 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:47.779+0000 7f42ba0d5040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T12:20:49.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:48 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:48.684+0000 7f42ba0d5040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T12:20:49.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:48 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:48.751+0000 7f42ba0d5040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T12:20:49.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:48 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:48.824+0000 7fbf71a13040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T12:20:49.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:48 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:48.904+0000 7fbf71a13040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T12:20:49.603 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:49 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:49.479+0000 7f42ba0d5040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T12:20:49.603 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:49 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:49.601+0000 7f42ba0d5040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T12:20:49.777 INFO:teuthology.orchestra.run.smithi196.stdout:[client.1] 2024-09-17T12:20:49.777 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQAhdOlmcIcRLhAAC8FJJqe0NoeJ3HOADCgAfQ== 2024-09-17T12:20:49.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:49 smithi016 ceph-mon[23719]: from='client.? 172.21.15.196:0/1063149691' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T12:20:49.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:49 smithi016 ceph-mon[23719]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T12:20:49.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:49 smithi016 ceph-mon[23719]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T12:20:49.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:49 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:49.672+0000 7fbf71a13040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T12:20:49.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:49 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:49.795+0000 7fbf71a13040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T12:20:49.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:49 smithi016 ceph-mon[34212]: from='client.? 172.21.15.196:0/1063149691' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T12:20:49.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:49 smithi016 ceph-mon[34212]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T12:20:49.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:49 smithi016 ceph-mon[34212]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T12:20:49.958 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:49 smithi196 ceph-mon[27428]: from='client.? 172.21.15.196:0/1063149691' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T12:20:49.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:49 smithi196 ceph-mon[27428]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T12:20:49.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:49 smithi196 ceph-mon[27428]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T12:20:49.959 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:49 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:49.745+0000 7f42ba0d5040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:20:49.959 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:49 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:49.811+0000 7f42ba0d5040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T12:20:49.959 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:49 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:49.957+0000 7f42ba0d5040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T12:20:50.240 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:49 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:49.946+0000 7fbf71a13040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:20:50.240 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:50.011+0000 7fbf71a13040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T12:20:50.240 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:50.166+0000 7fbf71a13040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T12:20:50.245 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:50 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:50.025+0000 7f42ba0d5040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T12:20:50.246 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:50 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:50.087+0000 7f42ba0d5040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T12:20:50.246 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:50 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:50.244+0000 7f42ba0d5040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T12:20:50.468 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-17T12:20:50.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-17T12:20:50.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-17T12:20:50.496 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:50 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:50.309+0000 7f42ba0d5040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T12:20:50.496 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:50 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:50.375+0000 7f42ba0d5040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T12:20:50.496 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:50 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:50.459+0000 7f42ba0d5040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T12:20:50.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph config log 1 --format=json 2024-09-17T12:20:50.532 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:50.237+0000 7fbf71a13040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T12:20:50.532 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:50.301+0000 7fbf71a13040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T12:20:50.532 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:50.465+0000 7fbf71a13040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T12:20:50.715 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:20:50.776 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:50 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:50.522+0000 7f42ba0d5040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T12:20:50.776 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:50 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:50.581+0000 7f42ba0d5040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T12:20:50.832 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:50.535+0000 7fbf71a13040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T12:20:50.832 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:50.607+0000 7fbf71a13040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T12:20:50.833 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:50.701+0000 7fbf71a13040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T12:20:50.833 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:50.767+0000 7fbf71a13040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T12:20:50.833 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:50 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:50.830+0000 7fbf71a13040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T12:20:51.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-mon[23719]: Standby manager daemon x restarted 2024-09-17T12:20:51.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-mon[23719]: Standby manager daemon x started 2024-09-17T12:20:51.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/3697695434' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:20:51.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/3697695434' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:20:51.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/3697695434' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:20:51.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/3697695434' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:20:51.676 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-mon[34212]: Standby manager daemon x restarted 2024-09-17T12:20:51.676 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-mon[34212]: Standby manager daemon x started 2024-09-17T12:20:51.676 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/3697695434' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:20:51.676 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/3697695434' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:20:51.676 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/3697695434' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:20:51.676 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/3697695434' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:20:51.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:51 smithi196 ceph-mon[27428]: Standby manager daemon x restarted 2024-09-17T12:20:51.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:51 smithi196 ceph-mon[27428]: Standby manager daemon x started 2024-09-17T12:20:51.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:51 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/3697695434' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:20:51.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:51 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/3697695434' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:20:51.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:51 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/3697695434' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:20:51.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:51 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/3697695434' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:20:51.777 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:51 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:51.351+0000 7f42ba0d5040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T12:20:51.777 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:51 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: [17/Sep/2024:12:20:51] ENGINE Bus STARTING 2024-09-17T12:20:51.777 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:51 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: CherryPy Checker: 2024-09-17T12:20:51.777 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:51 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: The Application mounted at '' has an empty config. 2024-09-17T12:20:51.777 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:51 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: 2024-09-17T12:20:51.778 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:51 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: [17/Sep/2024:12:20:51] ENGINE Serving on http://:::9283 2024-09-17T12:20:51.778 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:20:51 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[29266]: [17/Sep/2024:12:20:51] ENGINE Bus STARTED 2024-09-17T12:20:51.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:51.672+0000 7fbf71a13040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T12:20:51.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:20:51] ENGINE Bus STARTING 2024-09-17T12:20:51.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: CherryPy Checker: 2024-09-17T12:20:51.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: The Application mounted at '' has an empty config. 2024-09-17T12:20:51.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:51.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:20:51] ENGINE Serving on http://:::9283 2024-09-17T12:20:51.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:51 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:20:51] ENGINE Bus STARTED 2024-09-17T12:20:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-17T12:20:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: Active manager daemon y restarted 2024-09-17T12:20:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: Activating manager daemon y 2024-09-17T12:20:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: osdmap e61: 8 total, 8 up, 8 in 2024-09-17T12:20:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: mgrmap e18: y(active, starting, since 0.00724611s), standbys: x 2024-09-17T12:20:52.675 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:20:52.675 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:20:52.675 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:20:52.676 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:20:52.676 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:20:52.676 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:20:52.676 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:20:52.676 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:20:52.676 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:20:52.676 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:20:52.676 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:20:52.676 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:20:52.676 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:52.676 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:20:52.676 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:20:52.676 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:20:52.677 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: Manager daemon y is now available 2024-09-17T12:20:52.677 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:20:52.677 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:52.677 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:20:52.677 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:20:52.677 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:20:52.677 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:20:52.677 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:52 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:20:52.690 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:20:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-17T12:20:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: Active manager daemon y restarted 2024-09-17T12:20:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: Activating manager daemon y 2024-09-17T12:20:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: osdmap e61: 8 total, 8 up, 8 in 2024-09-17T12:20:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: mgrmap e18: y(active, starting, since 0.00724611s), standbys: x 2024-09-17T12:20:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:20:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:20:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:20:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:20:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:20:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:20:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:20:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:20:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:20:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:20:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:20:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:20:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:20:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:20:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:20:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: Manager daemon y is now available 2024-09-17T12:20:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:20:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:52.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:20:52.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:20:52.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:20:52.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:20:52.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:20:52.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-17T12:20:52.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: Active manager daemon y restarted 2024-09-17T12:20:52.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: Activating manager daemon y 2024-09-17T12:20:52.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: osdmap e61: 8 total, 8 up, 8 in 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: mgrmap e18: y(active, starting, since 0.00724611s), standbys: x 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:20:52.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:20:52.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:20:52.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:20:52.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:20:52.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: Manager daemon y is now available 2024-09-17T12:20:52.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:20:52.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:52.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:20:52.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:20:52.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:20:52.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:20:52.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:52 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:20:53.803 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:53 smithi196 ceph-mon[27428]: mgrmap e19: y(active, since 1.01245s), standbys: x 2024-09-17T12:20:53.803 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:53 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3195673264' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-17T12:20:53.803 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:53 smithi196 ceph-mon[27428]: [17/Sep/2024:12:20:53] ENGINE Bus STARTING 2024-09-17T12:20:53.803 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:53 smithi196 ceph-mon[27428]: [17/Sep/2024:12:20:53] ENGINE Serving on http://172.21.15.16:8765 2024-09-17T12:20:53.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-mon[23719]: mgrmap e19: y(active, since 1.01245s), standbys: x 2024-09-17T12:20:53.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3195673264' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-17T12:20:53.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-mon[23719]: [17/Sep/2024:12:20:53] ENGINE Bus STARTING 2024-09-17T12:20:53.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-mon[23719]: [17/Sep/2024:12:20:53] ENGINE Serving on http://172.21.15.16:8765 2024-09-17T12:20:53.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-mon[34212]: mgrmap e19: y(active, since 1.01245s), standbys: x 2024-09-17T12:20:53.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3195673264' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-17T12:20:53.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-mon[34212]: [17/Sep/2024:12:20:53] ENGINE Bus STARTING 2024-09-17T12:20:53.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-mon[34212]: [17/Sep/2024:12:20:53] ENGINE Serving on http://172.21.15.16:8765 2024-09-17T12:20:53.854 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:53.668+0000 7fbec9f82640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:12:20:53] ENGINE Error in HTTPServer.serve 2024-09-17T12:20:53.854 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: Traceback (most recent call last): 2024-09-17T12:20:53.854 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:20:53.854 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: self._connections.run(self.expiration_interval) 2024-09-17T12:20:53.854 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:20:53.854 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: self._run(expiration_interval) 2024-09-17T12:20:53.854 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:20:53.854 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:20:53.854 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:20:53.854 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:20:53.854 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:20:53.854 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: s = self.context.wrap_socket( 2024-09-17T12:20:53.855 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:20:53.855 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: return self.sslsocket_class._create( 2024-09-17T12:20:53.855 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:20:53.855 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: self.do_handshake() 2024-09-17T12:20:53.855 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:20:53.855 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: self._sslobj.do_handshake() 2024-09-17T12:20:53.855 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:20:53.855 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:20:53.873 INFO:teuthology.orchestra.run.smithi016.stdout:[{"version":23,"timestamp":"2024-09-17T12:20:23.270566+0000","name":"","changes":[{"name":"osd/host:smithi196/osd_memory_target","previous_value":"5972708147","new_value":"4479531110"}]}] 2024-09-17T12:20:53.873 INFO:tasks.ceph_manager:config epoch is 23 2024-09-17T12:20:53.873 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T12:20:53.873 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-17T12:20:53.874 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-17T12:20:54.131 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:20:54.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: [17/Sep/2024:12:20:53] ENGINE Serving on https://172.21.15.16:7150 2024-09-17T12:20:54.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: [17/Sep/2024:12:20:53] ENGINE Bus STARTED 2024-09-17T12:20:54.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: [17/Sep/2024:12:20:53] ENGINE Error in HTTPServer.serve 2024-09-17T12:20:54.942 INFO:journalctl@ceph.mon.c.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:20:54.942 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:20:54.942 INFO:journalctl@ceph.mon.c.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:20:54.942 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:20:54.942 INFO:journalctl@ceph.mon.c.smithi016.stdout: self._run(expiration_interval) 2024-09-17T12:20:54.942 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:20:54.942 INFO:journalctl@ceph.mon.c.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:20:54.942 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:20:54.942 INFO:journalctl@ceph.mon.c.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:20:54.942 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:20:54.943 INFO:journalctl@ceph.mon.c.smithi016.stdout: s = self.context.wrap_socket( 2024-09-17T12:20:54.943 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:20:54.943 INFO:journalctl@ceph.mon.c.smithi016.stdout: return self.sslsocket_class._create( 2024-09-17T12:20:54.943 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:20:54.943 INFO:journalctl@ceph.mon.c.smithi016.stdout: self.do_handshake() 2024-09-17T12:20:54.943 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:20:54.943 INFO:journalctl@ceph.mon.c.smithi016.stdout: self._sslobj.do_handshake() 2024-09-17T12:20:54.943 INFO:journalctl@ceph.mon.c.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:20:54.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-17T12:20:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: Adjusting osd_memory_target on smithi196 to 4016M 2024-09-17T12:20:54.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:54.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: [17/Sep/2024:12:20:53] ENGINE Serving on https://172.21.15.16:7150 2024-09-17T12:20:54.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: [17/Sep/2024:12:20:53] ENGINE Bus STARTED 2024-09-17T12:20:54.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: [17/Sep/2024:12:20:53] ENGINE Error in HTTPServer.serve 2024-09-17T12:20:54.945 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:20:54.945 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:20:54.946 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:20:54.946 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:20:54.946 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-09-17T12:20:54.946 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:20:54.946 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:20:54.946 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:20:54.946 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:20:54.946 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:20:54.946 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-09-17T12:20:54.946 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:20:54.946 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-09-17T12:20:54.946 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:20:54.946 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:54.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: Adjusting osd_memory_target on smithi196 to 4016M 2024-09-17T12:20:54.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: [17/Sep/2024:12:20:53] ENGINE Serving on https://172.21.15.16:7150 2024-09-17T12:20:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: [17/Sep/2024:12:20:53] ENGINE Bus STARTED 2024-09-17T12:20:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: [17/Sep/2024:12:20:53] ENGINE Error in HTTPServer.serve 2024-09-17T12:20:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-17T12:20:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:20:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:20:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:20:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._run(expiration_interval) 2024-09-17T12:20:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:20:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:20:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:20:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:20:55.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:20:55.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: s = self.context.wrap_socket( 2024-09-17T12:20:55.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:20:55.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.sslsocket_class._create( 2024-09-17T12:20:55.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:20:55.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.do_handshake() 2024-09-17T12:20:55.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:20:55.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._sslobj.do_handshake() 2024-09-17T12:20:55.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:20:55.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:55.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-17T12:20:55.031 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.031 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.031 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.031 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.031 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.032 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.032 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:55.032 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:55.032 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:55.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:55.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:55.034 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:55.034 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:55.034 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:20:55.034 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: Adjusting osd_memory_target on smithi196 to 4016M 2024-09-17T12:20:55.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:55.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:20:55.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:20:55.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:20:55.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:20:55.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:20:55.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:20:55.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:20:55.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:20:55.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:20:55.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:55.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:20:55.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:20:55.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:20:55.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:20:55.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:20:55.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:20:55.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:20:55.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:20:55.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:20:55.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:55.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:55 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:20:56.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:20:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:20:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:20:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:20:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:20:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:20:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:20:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:20:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:20:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:20:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:56.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:56.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:56.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:56.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:55 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:20:56.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:20:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:20:56] "GET /metrics HTTP/1.1" 200 23675 "" "Prometheus/2.43.0" 2024-09-17T12:20:56.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:56 smithi016 ceph-mon[34212]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:56.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:56 smithi016 ceph-mon[34212]: Deploying daemon alertmanager.a on smithi016 2024-09-17T12:20:56.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:56 smithi016 ceph-mon[34212]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-17T12:20:56.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:56 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3310700739' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T12:20:56.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:56 smithi016 ceph-mon[23719]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:56.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:56 smithi016 ceph-mon[23719]: Deploying daemon alertmanager.a on smithi016 2024-09-17T12:20:56.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:56 smithi016 ceph-mon[23719]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-17T12:20:56.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:56 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3310700739' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T12:20:57.231 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"flags":0,"active_gid":14430,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2599403596},{"type":"v1","addr":"172.21.15.16:6801","nonce":2599403596}]},"active_addr":"172.21.15.16:6801/2599403596","active_change":"2024-09-17T12:20:51.675263+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24349,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.16:8443/","prometheus":"http://172.21.15.16:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":61,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":669007717}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1687402001}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3734045536}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1859821668}]}]} 2024-09-17T12:20:57.235 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-17T12:20:57.236 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-17T12:20:57.236 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T12:20:57.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:56 smithi196 ceph-mon[27428]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:57.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:56 smithi196 ceph-mon[27428]: Deploying daemon alertmanager.a on smithi016 2024-09-17T12:20:57.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:56 smithi196 ceph-mon[27428]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-17T12:20:57.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:56 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3310700739' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T12:20:57.446 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:20:59.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:20:58 smithi016 ceph-mon[23719]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:59.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:20:58 smithi016 ceph-mon[34212]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:20:59.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:20:58 smithi196 ceph-mon[27428]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:00.640 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:21:00.641 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"fsid":"5b9f4610-74ee-11ef-bceb-c7b262605968","created":"2024-09-17T12:15:25.688415+0000","modified":"2024-09-17T12:20:51.675132+0000","last_up_change":"2024-09-17T12:20:21.089433+0000","last_in_change":"2024-09-17T12:20:04.339833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T12:18:24.896429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32f1f69d-aaf7-42ed-92f0-9d1aa8bdd3c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":869442370},{"type":"v1","addr":"172.21.15.16:6803","nonce":869442370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":869442370},{"type":"v1","addr":"172.21.15.16:6805","nonce":869442370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":869442370},{"type":"v1","addr":"172.21.15.16:6809","nonce":869442370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":869442370},{"type":"v1","addr":"172.21.15.16:6807","nonce":869442370}]},"public_addr":"172.21.15.16:6803/869442370","cluster_addr":"172.21.15.16:6805/869442370","heartbeat_back_addr":"172.21.15.16:6809/869442370","heartbeat_front_addr":"172.21.15.16:6807/869442370","state":["exists","up"]},{"osd":1,"uuid":"b2dc9d4d-bcc8-4492-af56-57caea2ec732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":755127111},{"type":"v1","addr":"172.21.15.16:6811","nonce":755127111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":755127111},{"type":"v1","addr":"172.21.15.16:6813","nonce":755127111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":755127111},{"type":"v1","addr":"172.21.15.16:6817","nonce":755127111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":755127111},{"type":"v1","addr":"172.21.15.16:6815","nonce":755127111}]},"public_addr":"172.21.15.16:6811/755127111","cluster_addr":"172.21.15.16:6813/755127111","heartbeat_back_addr":"172.21.15.16:6817/755127111","heartbeat_front_addr":"172.21.15.16:6815/755127111","state":["exists","up"]},{"osd":2,"uuid":"aa7dc4a4-7b6f-440c-bdd8-78f7f1786434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2313749493},{"type":"v1","addr":"172.21.15.16:6819","nonce":2313749493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2313749493},{"type":"v1","addr":"172.21.15.16:6821","nonce":2313749493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2313749493},{"type":"v1","addr":"172.21.15.16:6825","nonce":2313749493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2313749493},{"type":"v1","addr":"172.21.15.16:6823","nonce":2313749493}]},"public_addr":"172.21.15.16:6819/2313749493","cluster_addr":"172.21.15.16:6821/2313749493","heartbeat_back_addr":"172.21.15.16:6825/2313749493","heartbeat_front_addr":"172.21.15.16:6823/2313749493","state":["exists","up"]},{"osd":3,"uuid":"81f6d250-32a6-4276-b6bc-aac7e27e1a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3579869089},{"type":"v1","addr":"172.21.15.16:6827","nonce":3579869089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3579869089},{"type":"v1","addr":"172.21.15.16:6829","nonce":3579869089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3579869089},{"type":"v1","addr":"172.21.15.16:6833","nonce":3579869089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3579869089},{"type":"v1","addr":"172.21.15.16:6831","nonce":3579869089}]},"public_addr":"172.21.15.16:6827/3579869089","cluster_addr":"172.21.15.16:6829/3579869089","heartbeat_back_addr":"172.21.15.16:6833/3579869089","heartbeat_front_addr":"172.21.15.16:6831/3579869089","state":["exists","up"]},{"osd":4,"uuid":"e6198e3b-3a1d-40a0-b3e8-9d34fc42ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":590253655},{"type":"v1","addr":"172.21.15.196:6801","nonce":590253655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":590253655},{"type":"v1","addr":"172.21.15.196:6803","nonce":590253655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":590253655},{"type":"v1","addr":"172.21.15.196:6807","nonce":590253655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":590253655},{"type":"v1","addr":"172.21.15.196:6805","nonce":590253655}]},"public_addr":"172.21.15.196:6801/590253655","cluster_addr":"172.21.15.196:6803/590253655","heartbeat_back_addr":"172.21.15.196:6807/590253655","heartbeat_front_addr":"172.21.15.196:6805/590253655","state":["exists","up"]},{"osd":5,"uuid":"7eaa3677-35e9-4ce5-8515-ca7a27286824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2204973118},{"type":"v1","addr":"172.21.15.196:6809","nonce":2204973118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2204973118},{"type":"v1","addr":"172.21.15.196:6811","nonce":2204973118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2204973118},{"type":"v1","addr":"172.21.15.196:6815","nonce":2204973118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2204973118},{"type":"v1","addr":"172.21.15.196:6813","nonce":2204973118}]},"public_addr":"172.21.15.196:6809/2204973118","cluster_addr":"172.21.15.196:6811/2204973118","heartbeat_back_addr":"172.21.15.196:6815/2204973118","heartbeat_front_addr":"172.21.15.196:6813/2204973118","state":["exists","up"]},{"osd":6,"uuid":"f7891e15-85f5-4729-9143-e34199f6c03e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3128478817},{"type":"v1","addr":"172.21.15.196:6817","nonce":3128478817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3128478817},{"type":"v1","addr":"172.21.15.196:6819","nonce":3128478817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3128478817},{"type":"v1","addr":"172.21.15.196:6823","nonce":3128478817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3128478817},{"type":"v1","addr":"172.21.15.196:6821","nonce":3128478817}]},"public_addr":"172.21.15.196:6817/3128478817","cluster_addr":"172.21.15.196:6819/3128478817","heartbeat_back_addr":"172.21.15.196:6823/3128478817","heartbeat_front_addr":"172.21.15.196:6821/3128478817","state":["exists","up"]},{"osd":7,"uuid":"5698cae8-7738-4933-a108-4b039e629641","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":827333513},{"type":"v1","addr":"172.21.15.196:6825","nonce":827333513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":827333513},{"type":"v1","addr":"172.21.15.196:6827","nonce":827333513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":827333513},{"type":"v1","addr":"172.21.15.196:6831","nonce":827333513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":827333513},{"type":"v1","addr":"172.21.15.196:6829","nonce":827333513}]},"public_addr":"172.21.15.196:6825/827333513","cluster_addr":"172.21.15.196:6827/827333513","heartbeat_back_addr":"172.21.15.196:6831/827333513","heartbeat_front_addr":"172.21.15.196:6829/827333513","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:17:33.036896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:17:56.357381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:18:20.441837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:18:44.476130+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:19:06.358889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:19:30.976024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:19:55.394849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:20:19.670476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3211434426":"2024-09-18T12:20:51.675101+0000","172.21.15.16:0/1225259398":"2024-09-18T12:20:51.675101+0000","172.21.15.16:0/1310624606":"2024-09-18T12:15:52.139948+0000","172.21.15.16:0/843792520":"2024-09-18T12:15:52.139948+0000","172.21.15.16:0/3411803517":"2024-09-18T12:16:14.402306+0000","172.21.15.16:0/1179446893":"2024-09-18T12:20:51.675101+0000","172.21.15.16:6801/3212434883":"2024-09-18T12:15:52.139948+0000","172.21.15.16:6800/3212434883":"2024-09-18T12:15:52.139948+0000","172.21.15.16:0/29394240":"2024-09-18T12:16:14.402306+0000","172.21.15.16:6800/4014507619":"2024-09-18T12:16:14.402306+0000","172.21.15.16:6800/3754043720":"2024-09-18T12:20:51.675101+0000","172.21.15.16:0/218077182":"2024-09-18T12:16:14.402306+0000","172.21.15.16:0/610303924":"2024-09-18T12:15:52.139948+0000","172.21.15.16:0/3785112609":"2024-09-18T12:20:51.675101+0000","172.21.15.16:6801/4014507619":"2024-09-18T12:16:14.402306+0000","172.21.15.16:6801/3754043720":"2024-09-18T12:20:51.675101+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T12:21:01.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:00 smithi016 ceph-mon[23719]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T12:21:01.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:00 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1145511792' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T12:21:01.177 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:00 smithi016 ceph-mon[34212]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T12:21:01.178 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:00 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1145511792' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T12:21:01.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:00 smithi196 ceph-mon[27428]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T12:21:01.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:00 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1145511792' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T12:21:02.097 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-17T12:21:02.098 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T12:21:03.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:02 smithi016 ceph-mon[34212]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T12:21:03.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:02 smithi016 ceph-mon[23719]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T12:21:03.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:02 smithi196 ceph-mon[27428]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T12:21:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:05.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:04 smithi016 ceph-mon[23719]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T12:21:05.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:04 smithi016 ceph-mon[34212]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T12:21:05.158 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:05 smithi016 systemd[1]: Starting Ceph alertmanager.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:21:05.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:04 smithi196 ceph-mon[27428]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T12:21:05.686 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:21:05.686 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"fsid":"5b9f4610-74ee-11ef-bceb-c7b262605968","created":"2024-09-17T12:15:25.688415+0000","modified":"2024-09-17T12:20:51.675132+0000","last_up_change":"2024-09-17T12:20:21.089433+0000","last_in_change":"2024-09-17T12:20:04.339833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T12:18:24.896429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32f1f69d-aaf7-42ed-92f0-9d1aa8bdd3c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":869442370},{"type":"v1","addr":"172.21.15.16:6803","nonce":869442370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":869442370},{"type":"v1","addr":"172.21.15.16:6805","nonce":869442370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":869442370},{"type":"v1","addr":"172.21.15.16:6809","nonce":869442370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":869442370},{"type":"v1","addr":"172.21.15.16:6807","nonce":869442370}]},"public_addr":"172.21.15.16:6803/869442370","cluster_addr":"172.21.15.16:6805/869442370","heartbeat_back_addr":"172.21.15.16:6809/869442370","heartbeat_front_addr":"172.21.15.16:6807/869442370","state":["exists","up"]},{"osd":1,"uuid":"b2dc9d4d-bcc8-4492-af56-57caea2ec732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":755127111},{"type":"v1","addr":"172.21.15.16:6811","nonce":755127111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":755127111},{"type":"v1","addr":"172.21.15.16:6813","nonce":755127111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":755127111},{"type":"v1","addr":"172.21.15.16:6817","nonce":755127111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":755127111},{"type":"v1","addr":"172.21.15.16:6815","nonce":755127111}]},"public_addr":"172.21.15.16:6811/755127111","cluster_addr":"172.21.15.16:6813/755127111","heartbeat_back_addr":"172.21.15.16:6817/755127111","heartbeat_front_addr":"172.21.15.16:6815/755127111","state":["exists","up"]},{"osd":2,"uuid":"aa7dc4a4-7b6f-440c-bdd8-78f7f1786434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2313749493},{"type":"v1","addr":"172.21.15.16:6819","nonce":2313749493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2313749493},{"type":"v1","addr":"172.21.15.16:6821","nonce":2313749493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2313749493},{"type":"v1","addr":"172.21.15.16:6825","nonce":2313749493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2313749493},{"type":"v1","addr":"172.21.15.16:6823","nonce":2313749493}]},"public_addr":"172.21.15.16:6819/2313749493","cluster_addr":"172.21.15.16:6821/2313749493","heartbeat_back_addr":"172.21.15.16:6825/2313749493","heartbeat_front_addr":"172.21.15.16:6823/2313749493","state":["exists","up"]},{"osd":3,"uuid":"81f6d250-32a6-4276-b6bc-aac7e27e1a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3579869089},{"type":"v1","addr":"172.21.15.16:6827","nonce":3579869089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3579869089},{"type":"v1","addr":"172.21.15.16:6829","nonce":3579869089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3579869089},{"type":"v1","addr":"172.21.15.16:6833","nonce":3579869089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3579869089},{"type":"v1","addr":"172.21.15.16:6831","nonce":3579869089}]},"public_addr":"172.21.15.16:6827/3579869089","cluster_addr":"172.21.15.16:6829/3579869089","heartbeat_back_addr":"172.21.15.16:6833/3579869089","heartbeat_front_addr":"172.21.15.16:6831/3579869089","state":["exists","up"]},{"osd":4,"uuid":"e6198e3b-3a1d-40a0-b3e8-9d34fc42ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":590253655},{"type":"v1","addr":"172.21.15.196:6801","nonce":590253655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":590253655},{"type":"v1","addr":"172.21.15.196:6803","nonce":590253655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":590253655},{"type":"v1","addr":"172.21.15.196:6807","nonce":590253655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":590253655},{"type":"v1","addr":"172.21.15.196:6805","nonce":590253655}]},"public_addr":"172.21.15.196:6801/590253655","cluster_addr":"172.21.15.196:6803/590253655","heartbeat_back_addr":"172.21.15.196:6807/590253655","heartbeat_front_addr":"172.21.15.196:6805/590253655","state":["exists","up"]},{"osd":5,"uuid":"7eaa3677-35e9-4ce5-8515-ca7a27286824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2204973118},{"type":"v1","addr":"172.21.15.196:6809","nonce":2204973118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2204973118},{"type":"v1","addr":"172.21.15.196:6811","nonce":2204973118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2204973118},{"type":"v1","addr":"172.21.15.196:6815","nonce":2204973118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2204973118},{"type":"v1","addr":"172.21.15.196:6813","nonce":2204973118}]},"public_addr":"172.21.15.196:6809/2204973118","cluster_addr":"172.21.15.196:6811/2204973118","heartbeat_back_addr":"172.21.15.196:6815/2204973118","heartbeat_front_addr":"172.21.15.196:6813/2204973118","state":["exists","up"]},{"osd":6,"uuid":"f7891e15-85f5-4729-9143-e34199f6c03e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3128478817},{"type":"v1","addr":"172.21.15.196:6817","nonce":3128478817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3128478817},{"type":"v1","addr":"172.21.15.196:6819","nonce":3128478817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3128478817},{"type":"v1","addr":"172.21.15.196:6823","nonce":3128478817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3128478817},{"type":"v1","addr":"172.21.15.196:6821","nonce":3128478817}]},"public_addr":"172.21.15.196:6817/3128478817","cluster_addr":"172.21.15.196:6819/3128478817","heartbeat_back_addr":"172.21.15.196:6823/3128478817","heartbeat_front_addr":"172.21.15.196:6821/3128478817","state":["exists","up"]},{"osd":7,"uuid":"5698cae8-7738-4933-a108-4b039e629641","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":827333513},{"type":"v1","addr":"172.21.15.196:6825","nonce":827333513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":827333513},{"type":"v1","addr":"172.21.15.196:6827","nonce":827333513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":827333513},{"type":"v1","addr":"172.21.15.196:6831","nonce":827333513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":827333513},{"type":"v1","addr":"172.21.15.196:6829","nonce":827333513}]},"public_addr":"172.21.15.196:6825/827333513","cluster_addr":"172.21.15.196:6827/827333513","heartbeat_back_addr":"172.21.15.196:6831/827333513","heartbeat_front_addr":"172.21.15.196:6829/827333513","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:17:33.036896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:17:56.357381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:18:20.441837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:18:44.476130+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:19:06.358889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:19:30.976024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:19:55.394849+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T12:20:19.670476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3211434426":"2024-09-18T12:20:51.675101+0000","172.21.15.16:0/1225259398":"2024-09-18T12:20:51.675101+0000","172.21.15.16:0/1310624606":"2024-09-18T12:15:52.139948+0000","172.21.15.16:0/843792520":"2024-09-18T12:15:52.139948+0000","172.21.15.16:0/3411803517":"2024-09-18T12:16:14.402306+0000","172.21.15.16:0/1179446893":"2024-09-18T12:20:51.675101+0000","172.21.15.16:6801/3212434883":"2024-09-18T12:15:52.139948+0000","172.21.15.16:6800/3212434883":"2024-09-18T12:15:52.139948+0000","172.21.15.16:0/29394240":"2024-09-18T12:16:14.402306+0000","172.21.15.16:6800/4014507619":"2024-09-18T12:16:14.402306+0000","172.21.15.16:6800/3754043720":"2024-09-18T12:20:51.675101+0000","172.21.15.16:0/218077182":"2024-09-18T12:16:14.402306+0000","172.21.15.16:0/610303924":"2024-09-18T12:15:52.139948+0000","172.21.15.16:0/3785112609":"2024-09-18T12:20:51.675101+0000","172.21.15.16:6801/4014507619":"2024-09-18T12:16:14.402306+0000","172.21.15.16:6801/3754043720":"2024-09-18T12:20:51.675101+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T12:21:05.686 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:05 smithi016 podman[63715]: 2024-09-17 12:21:05.549796451 +0000 UTC m=+0.023469735 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-17T12:21:05.687 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:05 smithi016 podman[63715]: 2024-09-17 12:21:05.650773401 +0000 UTC m=+0.124446670 volume create 7ad842b87448fa353a13cfa220db0c870b02d0a0904a89decb4cd866c8e5d8f1 2024-09-17T12:21:05.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:05 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/4127941933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T12:21:05.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:05 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/4127941933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T12:21:05.942 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:05 smithi016 podman[63715]: 2024-09-17 12:21:05.745295321 +0000 UTC m=+0.218968590 container create 83f08a029887f9669dca2893a4dea1b604447a544afa224542ca1f0d5b8abd82 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:06.200 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:06 smithi016 podman[63715]: 2024-09-17 12:21:06.181957338 +0000 UTC m=+0.655630609 container init 83f08a029887f9669dca2893a4dea1b604447a544afa224542ca1f0d5b8abd82 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:06.200 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:06 smithi016 podman[63715]: 2024-09-17 12:21:06.185073678 +0000 UTC m=+0.658746946 container start 83f08a029887f9669dca2893a4dea1b604447a544afa224542ca1f0d5b8abd82 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:06.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:05 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/4127941933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T12:21:06.462 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:21:06] "GET /metrics HTTP/1.1" 200 23675 "" "Prometheus/2.43.0" 2024-09-17T12:21:06.462 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[63742]: ts=2024-09-17T12:21:06.199Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-17T12:21:06.462 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[63742]: ts=2024-09-17T12:21:06.199Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-17T12:21:06.463 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[63742]: ts=2024-09-17T12:21:06.200Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.16 port=9094 2024-09-17T12:21:06.463 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[63742]: ts=2024-09-17T12:21:06.200Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T12:21:06.463 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[63742]: ts=2024-09-17T12:21:06.223Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T12:21:06.463 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[63742]: ts=2024-09-17T12:21:06.224Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T12:21:06.463 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[63742]: ts=2024-09-17T12:21:06.227Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-17T12:21:06.463 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[63742]: ts=2024-09-17T12:21:06.227Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-17T12:21:06.463 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:06 smithi016 bash[63715]: 83f08a029887f9669dca2893a4dea1b604447a544afa224542ca1f0d5b8abd82 2024-09-17T12:21:06.463 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:06 smithi016 systemd[1]: Started Ceph alertmanager.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:21:06.718 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:06] ENGINE Bus STOPPING 2024-09-17T12:21:06.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-17T12:21:06.876 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-17T12:21:06.876 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-17T12:21:06.877 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-17T12:21:06.877 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-17T12:21:06.877 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-17T12:21:06.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-17T12:21:06.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-17T12:21:07.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T12:21:07.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:06] ENGINE Bus STOPPED 2024-09-17T12:21:07.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:06] ENGINE Bus STARTING 2024-09-17T12:21:07.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:06] ENGINE Serving on http://:::9283 2024-09-17T12:21:07.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:06] ENGINE Bus STARTED 2024-09-17T12:21:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:07.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T12:21:07.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T12:21:07.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T12:21:07.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T12:21:07.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: Deploying daemon grafana.a on smithi196 2024-09-17T12:21:07.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:21:07.592 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T12:21:07.592 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.592 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.592 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.592 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.592 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T12:21:07.592 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.592 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.592 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T12:21:07.592 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T12:21:07.592 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.593 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: Deploying daemon grafana.a on smithi196 2024-09-17T12:21:07.593 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.593 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:07 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:21:07.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T12:21:07.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T12:21:07.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T12:21:07.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T12:21:07.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: Deploying daemon grafana.a on smithi196 2024-09-17T12:21:07.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:07.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:07 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:21:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:07.942 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:07.942 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:07.942 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:08.374 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:08 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[63742]: ts=2024-09-17T12:21:08.201Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000233063s 2024-09-17T12:21:09.595 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:09 smithi016 ceph-mon[23719]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T12:21:09.596 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:09 smithi016 ceph-mon[34212]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T12:21:09.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:09 smithi196 ceph-mon[27428]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T12:21:11.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:11 smithi016 ceph-mon[23719]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T12:21:11.669 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:11 smithi016 ceph-mon[34212]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T12:21:11.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:11 smithi196 ceph-mon[27428]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T12:21:13.287 INFO:teuthology.orchestra.run.smithi016.stdout:167503724565 2024-09-17T12:21:13.288 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-17T12:21:13.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:13 smithi016 ceph-mon[23719]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:13.566 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:13 smithi016 ceph-mon[34212]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:13.759 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:13 smithi196 ceph-mon[27428]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:15.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:15 smithi196 ceph-mon[27428]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:15.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:15 smithi016 ceph-mon[23719]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:15.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:15 smithi016 ceph-mon[34212]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:16.329 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:21:16] "GET /metrics HTTP/1.1" 200 26201 "" "Prometheus/2.43.0" 2024-09-17T12:21:16.363 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[63742]: ts=2024-09-17T12:21:16.204Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003965193s 2024-09-17T12:21:16.527 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 systemd[1]: Starting Ceph grafana.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:21:16.945 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 podman[56299]: 2024-09-17 12:21:16.529997597 +0000 UTC m=+0.015405866 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-17T12:21:16.945 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 podman[56299]: 2024-09-17 12:21:16.641144007 +0000 UTC m=+0.126552257 container create 52809a2352add56bc6fb2a5742dedce816c71a1423289bcf0b67dc99fd21619d (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, name=ubi8, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, description=Ceph Grafana Container, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, version=8.5, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , release=239.1651231664) 2024-09-17T12:21:16.945 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 podman[56299]: 2024-09-17 12:21:16.843308008 +0000 UTC m=+0.328716270 container init 52809a2352add56bc6fb2a5742dedce816c71a1423289bcf0b67dc99fd21619d (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, com.redhat.component=ubi8-container, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, name=ubi8, architecture=x86_64, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, maintainer=Paul Cuzner , io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-09-17T12:21:16.946 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 podman[56299]: 2024-09-17 12:21:16.84602866 +0000 UTC m=+0.331436903 container start 52809a2352add56bc6fb2a5742dedce816c71a1423289bcf0b67dc99fd21619d (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, io.openshift.tags=base rhel8, release=239.1651231664, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, vendor=Red Hat, Inc., description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , io.openshift.expose-services=, distribution-scope=public) 2024-09-17T12:21:16.946 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 bash[56299]: 52809a2352add56bc6fb2a5742dedce816c71a1423289bcf0b67dc99fd21619d 2024-09-17T12:21:17.195 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 systemd[1]: Started Ceph grafana.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:21:17.195 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998061914Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-17T12:21:17.196 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998273781Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-17T12:21:17.196 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998292829Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-17T12:21:17.196 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998300716Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-17T12:21:17.196 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998306875Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-17T12:21:17.196 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998312927Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-17T12:21:17.197 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998318719Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-17T12:21:17.197 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998324644Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-17T12:21:17.197 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998330544Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-17T12:21:17.197 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998353306Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-17T12:21:17.197 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998360777Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-17T12:21:17.197 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998366307Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-17T12:21:17.198 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998372037Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-17T12:21:17.198 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=settings t=2024-09-17T12:21:16.998377878Z level=info msg="App mode production" 2024-09-17T12:21:17.198 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=sqlstore t=2024-09-17T12:21:16.998428301Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-17T12:21:17.198 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=sqlstore t=2024-09-17T12:21:16.998445852Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-17T12:21:17.198 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:16.999329983Z level=info msg="Starting DB migrations" 2024-09-17T12:21:17.199 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:16.999404036Z level=info msg="Executing migration" id="create migration_log table" 2024-09-17T12:21:17.199 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:16.999954059Z level=info msg="Executing migration" id="create user table" 2024-09-17T12:21:17.199 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.000430895Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-17T12:21:17.199 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.001098217Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-17T12:21:17.199 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.001545062Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-17T12:21:17.200 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.001991688Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-17T12:21:17.200 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.002370465Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-17T12:21:17.200 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.003073848Z level=info msg="Executing migration" id="create user table v2" 2024-09-17T12:21:17.200 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.003462142Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-17T12:21:17.200 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.003820008Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-17T12:21:17.200 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.004250986Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-17T12:21:17.201 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.004595833Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-17T12:21:17.201 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.004978821Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-17T12:21:17.201 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.005404192Z level=info msg="Executing migration" id="Update user table charset" 2024-09-17T12:21:17.201 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.005676704Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-17T12:21:17.201 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.006113835Z level=info msg="Executing migration" id="Add missing user data" 2024-09-17T12:21:17.201 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.006492854Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-17T12:21:17.201 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.007429345Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-17T12:21:17.201 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.007854643Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-17T12:21:17.202 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.008310625Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-17T12:21:17.202 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.00950886Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-17T12:21:17.203 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.009956284Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-17T12:21:17.203 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.010432066Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-17T12:21:17.203 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.010849597Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-17T12:21:17.203 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.011267038Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-17T12:21:17.203 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.011940948Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-17T12:21:17.203 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.012727157Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-17T12:21:17.204 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.0131182Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-17T12:21:17.204 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.013459008Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-17T12:21:17.204 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.01380821Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-17T12:21:17.204 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.014179021Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-17T12:21:17.204 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.014778453Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-17T12:21:17.204 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.015193004Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-17T12:21:17.204 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.015549047Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-17T12:21:17.204 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.015991539Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-17T12:21:17.204 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.016898747Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-17T12:21:17.204 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.017399738Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-17T12:21:17.204 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.017699628Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-17T12:21:17.204 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.018040725Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-17T12:21:17.205 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.018377029Z level=info msg="Executing migration" id="create star table" 2024-09-17T12:21:17.205 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.018706237Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-17T12:21:17.205 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.019139773Z level=info msg="Executing migration" id="create org table v1" 2024-09-17T12:21:17.205 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.019600604Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-17T12:21:17.205 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.020496598Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-17T12:21:17.205 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.020850663Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-17T12:21:17.205 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.021235939Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-17T12:21:17.205 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.021617559Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-17T12:21:17.205 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.022050951Z level=info msg="Executing migration" id="Update org table charset" 2024-09-17T12:21:17.205 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.022868314Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-17T12:21:17.206 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.023720583Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-17T12:21:17.206 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.024581485Z level=info msg="Executing migration" id="create dashboard table" 2024-09-17T12:21:17.206 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.025405568Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-17T12:21:17.206 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.02626396Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-17T12:21:17.206 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.02724806Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-17T12:21:17.206 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.027910652Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-17T12:21:17.206 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.028793435Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-17T12:21:17.206 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.029733865Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-17T12:21:17.206 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.03063797Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-17T12:21:17.206 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.031476309Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-17T12:21:17.206 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.032456618Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-17T12:21:17.206 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.033237591Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-17T12:21:17.207 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.034057619Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-17T12:21:17.207 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.034913779Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-17T12:21:17.207 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.035752137Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-17T12:21:17.207 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.036612112Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-17T12:21:17.207 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.037498483Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-17T12:21:17.207 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.038351602Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-17T12:21:17.207 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.039289964Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-17T12:21:17.207 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.040072531Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-17T12:21:17.207 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.04090985Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-17T12:21:17.207 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.041784628Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-17T12:21:17.207 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.042540195Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-17T12:21:17.207 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.043404456Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-17T12:21:17.208 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.043932009Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-17T12:21:17.208 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.044435871Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-17T12:21:17.208 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.045275502Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-17T12:21:17.208 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.046132256Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-17T12:21:17.208 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.048971147Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-17T12:21:17.208 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.05201893Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-17T12:21:17.208 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.054862004Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-17T12:21:17.208 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.057703524Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-17T12:21:17.208 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.060324204Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-17T12:21:17.208 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.063470013Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-17T12:21:17.208 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.066312053Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-17T12:21:17.208 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.06895964Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-17T12:21:17.209 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.069396045Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-17T12:21:17.209 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.070200521Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-17T12:21:17.209 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.071166812Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-17T12:21:17.209 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.07197106Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-17T12:21:17.209 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.072809538Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-17T12:21:17.209 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.073785157Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-17T12:21:17.209 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.074513058Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-17T12:21:17.209 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.075443276Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-17T12:21:17.209 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.075882971Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-17T12:21:17.209 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.078450795Z level=info msg="Executing migration" id="create data_source table" 2024-09-17T12:21:17.209 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.081056464Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-17T12:21:17.210 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.083989132Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-17T12:21:17.210 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.086610917Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-17T12:21:17.210 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.089509737Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-17T12:21:17.210 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.09214312Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-17T12:21:17.210 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.095047053Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-17T12:21:17.210 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.097859786Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-17T12:21:17.210 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.100645503Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-17T12:21:17.211 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.10349295Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-17T12:21:17.211 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.106373033Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-17T12:21:17.211 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.109342074Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-17T12:21:17.211 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.111878553Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-17T12:21:17.211 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.114611674Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-17T12:21:17.211 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.117264073Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-17T12:21:17.211 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.120118199Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-17T12:21:17.211 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.122922024Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-17T12:21:17.211 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.125584073Z level=info msg="Executing migration" id="Add uid column" 2024-09-17T12:21:17.211 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.128312722Z level=info msg="Executing migration" id="Update uid value" 2024-09-17T12:21:17.211 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.130844384Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-17T12:21:17.211 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.133518012Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-17T12:21:17.212 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.136126104Z level=info msg="Executing migration" id="create api_key table" 2024-09-17T12:21:17.212 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.138733021Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-17T12:21:17.212 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.141218999Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-17T12:21:17.212 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.144033836Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-17T12:21:17.212 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.146509231Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-17T12:21:17.212 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.148983896Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-17T12:21:17.212 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.149408904Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-17T12:21:17.212 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.151955906Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-17T12:21:17.212 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.153036222Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-17T12:21:17.212 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.154690997Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-17T12:21:17.212 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.157423341Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-17T12:21:17.212 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.158062903Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-17T12:21:17.213 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.160998793Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-17T12:21:17.213 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.161676398Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-17T12:21:17.213 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.162503069Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-17T12:21:17.213 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.162845101Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-17T12:21:17.213 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.163678725Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-17T12:21:17.213 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.164570158Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-17T12:21:17.213 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.165304211Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-17T12:21:17.213 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.165894842Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-17T12:21:17.213 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.166813724Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-17T12:21:17.213 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.167574694Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-17T12:21:17.214 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.168517355Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-17T12:21:17.214 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.16904991Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-17T12:21:17.214 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.169901415Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-17T12:21:17.214 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.170792537Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-17T12:21:17.214 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.171544008Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-17T12:21:17.214 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.171851027Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-17T12:21:17.214 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.172196638Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-17T12:21:17.214 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.172788469Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-17T12:21:17.214 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.173420721Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-17T12:21:17.215 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.173728694Z level=info msg="Executing migration" id="create quota table v1" 2024-09-17T12:21:17.215 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.174193017Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-17T12:21:17.215 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.174667687Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-17T12:21:17.215 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.17498818Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-17T12:21:17.215 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.175892958Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-17T12:21:17.215 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.176866659Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-17T12:21:17.215 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.177691176Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-17T12:21:17.215 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.178169186Z level=info msg="Executing migration" id="create session table" 2024-09-17T12:21:17.215 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.178988133Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-17T12:21:17.215 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.179921976Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-17T12:21:17.215 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.180655114Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-17T12:21:17.216 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.181063601Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-17T12:21:17.216 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.182048674Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-17T12:21:17.216 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.182774772Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-17T12:21:17.216 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.183586714Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-17T12:21:17.216 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.183896835Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-17T12:21:17.216 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.18496902Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-17T12:21:17.216 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.187876512Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-17T12:21:17.216 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.190445124Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-17T12:21:17.216 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.191049999Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-17T12:21:17.216 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.194000561Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-17T12:21:17.217 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.196629021Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-17T12:21:17.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-mon[27428]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:17.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:17.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:17.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:17.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:17.448 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.199393449Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-17T12:21:17.448 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.199741409Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-17T12:21:17.448 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.202631356Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-17T12:21:17.448 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.205427083Z level=info msg="Executing migration" id="create alert table v1" 2024-09-17T12:21:17.448 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.208255632Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-17T12:21:17.448 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.208700304Z level=info msg="Executing migration" id="add index alert state" 2024-09-17T12:21:17.448 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.211654487Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-17T12:21:17.448 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.214582009Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-17T12:21:17.448 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.217041327Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-17T12:21:17.448 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.217521939Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-17T12:21:17.448 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.220381346Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-17T12:21:17.449 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.223182022Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-17T12:21:17.449 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.22590356Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-17T12:21:17.449 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.226318878Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-17T12:21:17.449 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.228815607Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-17T12:21:17.449 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.229172233Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-17T12:21:17.449 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.231470157Z level=info msg="Executing migration" id="Add column is_default" 2024-09-17T12:21:17.449 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.232119588Z level=info msg="Executing migration" id="Add column frequency" 2024-09-17T12:21:17.449 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.234303178Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-17T12:21:17.449 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.235116056Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-17T12:21:17.449 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.237876958Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-17T12:21:17.449 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.240500557Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-17T12:21:17.449 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.243164849Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.243454632Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.246354792Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.24897957Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.251707008Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.252125927Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.254946208Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.255596695Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.2574108Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.258373956Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.258974705Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.259859296Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.260624801Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.261493558Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-17T12:21:17.450 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.261960256Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-17T12:21:17.451 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.262943357Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-17T12:21:17.451 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.263575856Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-17T12:21:17.451 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.264441595Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-17T12:21:17.451 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.26486249Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-17T12:21:17.451 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.265700354Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-17T12:21:17.451 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.266575904Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-17T12:21:17.451 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.267343792Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-17T12:21:17.451 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.267805305Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-17T12:21:17.451 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.268612787Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-17T12:21:17.451 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.269585704Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-17T12:21:17.451 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.270313368Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-17T12:21:17.451 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.270976628Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-17T12:21:17.451 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.27181821Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-17T12:21:17.452 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.272662288Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-17T12:21:17.452 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.273659607Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-17T12:21:17.452 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.275259814Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-17T12:21:17.452 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.278082559Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-17T12:21:17.452 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.280950019Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-17T12:21:17.452 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.283514489Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-17T12:21:17.452 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.28400079Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-17T12:21:17.453 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.286796609Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-17T12:21:17.453 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.289406744Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-17T12:21:17.453 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.292105065Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-17T12:21:17.454 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.292543641Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-17T12:21:17.454 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.295484327Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-17T12:21:17.454 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.298066053Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-17T12:21:17.454 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.300635151Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-17T12:21:17.454 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.301111304Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-17T12:21:17.454 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.302095567Z level=info msg="Executing migration" id="Move region to single row" 2024-09-17T12:21:17.454 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.302964547Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-17T12:21:17.454 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.303708268Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-17T12:21:17.454 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.304155196Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-17T12:21:17.454 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.304954953Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-17T12:21:17.454 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.305421689Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-17T12:21:17.455 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.305812173Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-17T12:21:17.455 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.306234484Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-17T12:21:17.455 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.306654754Z level=info msg="Executing migration" id="create test_data table" 2024-09-17T12:21:17.455 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.307076704Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-17T12:21:17.455 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.307483629Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-17T12:21:17.455 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.308382733Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-17T12:21:17.455 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.309100384Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-17T12:21:17.455 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.309876851Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-17T12:21:17.455 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.310220274Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-17T12:21:17.455 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.311028784Z level=info msg="Executing migration" id="create team table" 2024-09-17T12:21:17.455 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.311442688Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-17T12:21:17.455 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.311899069Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-17T12:21:17.455 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.312746707Z level=info msg="Executing migration" id="create team member table" 2024-09-17T12:21:17.456 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.313184905Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-17T12:21:17.456 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.314154122Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-17T12:21:17.456 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.314878047Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-17T12:21:17.456 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.31581368Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-17T12:21:17.456 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.316586328Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-17T12:21:17.456 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.317469197Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-17T12:21:17.456 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.318324192Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-17T12:21:17.456 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.319171684Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-17T12:21:17.456 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.319633794Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-17T12:21:17.456 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.32043523Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-17T12:21:17.456 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.320851088Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-17T12:21:17.456 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.321822088Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-17T12:21:17.456 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.322285321Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-17T12:21:17.457 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.323174262Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-17T12:21:17.457 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.324002591Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-17T12:21:17.457 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.324864864Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-17T12:21:17.457 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.325734271Z level=info msg="Executing migration" id="create tag table" 2024-09-17T12:21:17.457 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.328606163Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-17T12:21:17.457 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.331514312Z level=info msg="Executing migration" id="create login attempt table" 2024-09-17T12:21:17.457 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.334564927Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-17T12:21:17.457 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.337318566Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-17T12:21:17.457 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.340025893Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-17T12:21:17.457 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.342700347Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-17T12:21:17.457 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.345702704Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-17T12:21:17.457 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.348475547Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-17T12:21:17.457 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.350863889Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-17T12:21:17.458 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.353717145Z level=info msg="Executing migration" id="create user auth table" 2024-09-17T12:21:17.458 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.356599696Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-17T12:21:17.458 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.35943677Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-17T12:21:17.458 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.362240712Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-17T12:21:17.458 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.364945516Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-17T12:21:17.458 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.367643475Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-17T12:21:17.458 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.370551137Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-17T12:21:17.458 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.373286038Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-17T12:21:17.458 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.376151298Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-17T12:21:17.458 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.378815024Z level=info msg="Executing migration" id="create server_lock table" 2024-09-17T12:21:17.458 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.381437904Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-17T12:21:17.458 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.384064111Z level=info msg="Executing migration" id="create user auth token table" 2024-09-17T12:21:17.458 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.386728091Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-17T12:21:17.459 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.389417554Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-17T12:21:17.459 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.392151757Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-17T12:21:17.459 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.395011189Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-17T12:21:17.459 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.397670869Z level=info msg="Executing migration" id="create cache_data table" 2024-09-17T12:21:17.459 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.40020308Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-17T12:21:17.459 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.403002904Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-17T12:21:17.459 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.405601843Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-17T12:21:17.459 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.408194039Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-17T12:21:17.459 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.410947877Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-17T12:21:17.459 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.411461146Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-17T12:21:17.459 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.411955853Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-17T12:21:17.459 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.414678322Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-17T12:21:17.459 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.415202049Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-17T12:21:17.460 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.415489621Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-17T12:21:17.460 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.415927919Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-17T12:21:17.460 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.416386579Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-17T12:21:17.460 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.416808543Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-17T12:21:17.460 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.417298265Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-17T12:21:17.460 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.418153376Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-17T12:21:17.460 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.418595165Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-17T12:21:17.460 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.418897973Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-17T12:21:17.460 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.4193512Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-17T12:21:17.460 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.419770135Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-17T12:21:17.460 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.42022631Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-17T12:21:17.460 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.420535887Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.420992834Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.421422126Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.421973008Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.422443386Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.423365336Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.42382856Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.42431686Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.428013381Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.431437566Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.43190951Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.432345987Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.433198311Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.433693559Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-17T12:21:17.461 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.434147994Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-17T12:21:17.462 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.43462656Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-17T12:21:17.462 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.435068048Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-17T12:21:17.462 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.435375376Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-17T12:21:17.462 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.436212736Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-17T12:21:17.462 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.43702703Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-17T12:21:17.462 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.437843792Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-17T12:21:17.462 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.438290132Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-17T12:21:17.462 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.438713956Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-17T12:21:17.462 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.439521898Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-17T12:21:17.462 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.440361819Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-17T12:21:17.462 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.440816851Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-17T12:21:17.462 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.441637253Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-17T12:21:17.462 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.442492485Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-17T12:21:17.463 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.442825718Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-17T12:21:17.463 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.443302257Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-17T12:21:17.463 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.446071167Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-17T12:21:17.463 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.446580878Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-17T12:21:17.463 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.446913983Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-17T12:21:17.463 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.449464067Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-17T12:21:17.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:17 smithi016 ceph-mon[23719]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:17.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:17 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:17.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:17 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:17.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:17 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:17.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:17 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:17.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:17 smithi016 ceph-mon[34212]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:17.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:17 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:17.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:17 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:17.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:17 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:17.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:17 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:17.699 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.452187495Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-17T12:21:17.699 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.453050392Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-17T12:21:17.699 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.453881557Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-17T12:21:17.699 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.454786166Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-17T12:21:17.700 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.455614941Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-17T12:21:17.700 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.456461992Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-17T12:21:17.700 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.457353044Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-17T12:21:17.700 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.458231738Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-17T12:21:17.700 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.459131217Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-17T12:21:17.700 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.45996551Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-17T12:21:17.700 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.460848521Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-17T12:21:17.700 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.46178489Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-17T12:21:17.700 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.462506037Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-17T12:21:17.700 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.463382697Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-17T12:21:17.701 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.464278291Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-17T12:21:17.701 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.465036561Z level=info msg="Executing migration" id="create alert_image table" 2024-09-17T12:21:17.701 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.465891093Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-17T12:21:17.701 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.466821155Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-17T12:21:17.702 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.467722106Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-17T12:21:17.702 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.468500693Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-17T12:21:17.702 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.469265052Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-17T12:21:17.702 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.469369692Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-17T12:21:17.702 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.470270805Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-17T12:21:17.702 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.471032424Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-17T12:21:17.703 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.471915426Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-17T12:21:17.703 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.472044906Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-17T12:21:17.703 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.472178995Z level=info msg="alerts found to migrate" alerts=0 2024-09-17T12:21:17.703 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.474941647Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-17T12:21:17.703 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.477709378Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-17T12:21:17.703 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.480565173Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-17T12:21:17.703 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.483376217Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-17T12:21:17.703 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.486107043Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-17T12:21:17.703 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.488794061Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-17T12:21:17.703 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.491643024Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-17T12:21:17.703 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.494526726Z level=info msg="Executing migration" id="create data_keys table" 2024-09-17T12:21:17.703 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.497258274Z level=info msg="Executing migration" id="create secrets table" 2024-09-17T12:21:17.703 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.499820393Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-17T12:21:17.704 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.50409183Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-17T12:21:17.704 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.50698668Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-17T12:21:17.704 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.509967709Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-17T12:21:17.704 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.514387023Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-17T12:21:17.704 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.518596921Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-17T12:21:17.704 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.521211784Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-17T12:21:17.704 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.524021461Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-17T12:21:17.704 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.526851038Z level=info msg="Executing migration" id="create permission table" 2024-09-17T12:21:17.704 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.529499793Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-17T12:21:17.704 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.532185973Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-17T12:21:17.704 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.535024406Z level=info msg="Executing migration" id="create role table" 2024-09-17T12:21:17.704 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.537582003Z level=info msg="Executing migration" id="add column display_name" 2024-09-17T12:21:17.704 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.540212223Z level=info msg="Executing migration" id="add column group_name" 2024-09-17T12:21:17.705 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.542865897Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-17T12:21:17.705 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.545647077Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-17T12:21:17.705 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.54838822Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-17T12:21:17.705 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.550893228Z level=info msg="Executing migration" id="create team role table" 2024-09-17T12:21:17.705 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.553572315Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-17T12:21:17.705 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.556266247Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-17T12:21:17.705 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.559051952Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-17T12:21:17.705 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.561500001Z level=info msg="Executing migration" id="create user role table" 2024-09-17T12:21:17.705 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.564256957Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-17T12:21:17.705 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.567044614Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-17T12:21:17.705 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.569654439Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-17T12:21:17.705 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.572337613Z level=info msg="Executing migration" id="create builtin role table" 2024-09-17T12:21:17.705 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.573009717Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-17T12:21:17.706 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.573867911Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-17T12:21:17.706 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.574836098Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-17T12:21:17.706 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.575972296Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-17T12:21:17.706 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.576810397Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-17T12:21:17.706 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.577827425Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-17T12:21:17.706 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.578431644Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-17T12:21:17.706 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.57923001Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-17T12:21:17.706 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.57965835Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-17T12:21:17.706 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.580151559Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-17T12:21:17.706 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.581156795Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-17T12:21:17.706 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.581987375Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-17T12:21:17.706 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.5829773Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-17T12:21:17.706 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.583620123Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.584603933Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.585313926Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.586211986Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.586688025Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.587074543Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.587921721Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.58886129Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.589894321Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.590749754Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.591650181Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.592607565Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.593316629Z level=info msg="Executing migration" id="add correlation config column" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.59434981Z level=info msg="Executing migration" id="create entity_events table" 2024-09-17T12:21:17.707 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.595216898Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-17T12:21:17.708 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.596169107Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-17T12:21:17.708 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.596404021Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-17T12:21:17.708 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.596906983Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T12:21:17.708 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.597003443Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T12:21:17.708 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.597854962Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-17T12:21:17.708 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.598594608Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-17T12:21:17.708 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.599582078Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-17T12:21:17.708 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.600328871Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T12:21:17.708 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.601311339Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-17T12:21:17.708 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.604092941Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-17T12:21:17.708 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.606713054Z level=info msg="Executing migration" id="Drop public config table" 2024-09-17T12:21:17.708 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.609570202Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-17T12:21:17.708 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.612464771Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-17T12:21:17.709 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.614931467Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-17T12:21:17.709 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.617972362Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-17T12:21:17.709 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.620575984Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-17T12:21:17.709 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.623616825Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-17T12:21:17.709 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.626459417Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-17T12:21:17.709 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.629173587Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-17T12:21:17.709 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.631835388Z level=info msg="Executing migration" id="add share column" 2024-09-17T12:21:17.709 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.634663347Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-17T12:21:17.709 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.637458382Z level=info msg="Executing migration" id="create file table" 2024-09-17T12:21:17.709 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.640042811Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-17T12:21:17.709 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.642918492Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-17T12:21:17.709 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.645609859Z level=info msg="Executing migration" id="create file_meta table" 2024-09-17T12:21:17.709 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.648437059Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-17T12:21:17.710 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.651267551Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-17T12:21:17.710 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.654000157Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-17T12:21:17.710 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.656612165Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-17T12:21:17.710 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.659445701Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-17T12:21:17.710 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.662069214Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-17T12:21:17.710 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.664592981Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-17T12:21:17.710 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.66729652Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-17T12:21:17.710 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.670015221Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-17T12:21:17.710 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.672863057Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-17T12:21:17.710 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.675369755Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-17T12:21:17.710 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.677933706Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-17T12:21:17.710 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.680416782Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-17T12:21:17.710 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.683217643Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-17T12:21:17.711 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.685650258Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-17T12:21:17.711 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.696002026Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-17T12:21:17.711 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.698495145Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-17T12:21:17.711 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.701290644Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-17T12:21:18.019 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.704802987Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-17T12:21:18.019 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.705653751Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-17T12:21:18.019 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.706483467Z level=info msg="Executing migration" id="create folder table" 2024-09-17T12:21:18.020 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.707335444Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-17T12:21:18.020 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.707827978Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-17T12:21:18.020 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.708826879Z level=info msg="Executing migration" id="Update folder title length" 2024-09-17T12:21:18.020 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.709553033Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-17T12:21:18.020 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=migrator t=2024-09-17T12:21:17.710398196Z level=info msg="migrations completed" performed=485 skipped=0 duration=711.01373ms 2024-09-17T12:21:18.020 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=sqlstore t=2024-09-17T12:21:17.710914202Z level=info msg="Created default organization" 2024-09-17T12:21:18.020 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=plugin.loader t=2024-09-17T12:21:17.767985016Z level=info msg="Plugin registered" pluginID=input 2024-09-17T12:21:18.020 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=plugin.loader t=2024-09-17T12:21:17.794767398Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-17T12:21:18.020 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=plugin.loader t=2024-09-17T12:21:17.79478657Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-17T12:21:18.020 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=secrets t=2024-09-17T12:21:17.794819378Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-17T12:21:18.021 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=query_data t=2024-09-17T12:21:17.796450247Z level=info msg="Query Service initialization" 2024-09-17T12:21:18.021 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=live.push_http t=2024-09-17T12:21:17.798799386Z level=info msg="Live Push Gateway initialization" 2024-09-17T12:21:18.411 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:18 smithi016 ceph-mon[23719]: Deploying daemon node-exporter.a on smithi016 2024-09-17T12:21:18.411 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:18 smithi016 ceph-mon[34212]: Deploying daemon node-exporter.a on smithi016 2024-09-17T12:21:18.628 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:18 smithi196 ceph-mon[27428]: Deploying daemon node-exporter.a on smithi016 2024-09-17T12:21:18.628 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=infra.usagestats.collector t=2024-09-17T12:21:18.541366254Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-17T12:21:18.628 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=provisioning.datasources t=2024-09-17T12:21:18.595541055Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-17T12:21:18.628 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=provisioning.datasources t=2024-09-17T12:21:18.627274897Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-17T12:21:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:111669149727 2024-09-17T12:21:18.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-17T12:21:19.026 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=provisioning.alerting t=2024-09-17T12:21:18.649100092Z level=info msg="starting to provision alerting" 2024-09-17T12:21:19.027 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=provisioning.alerting t=2024-09-17T12:21:18.649144259Z level=info msg="finished to provision alerting" 2024-09-17T12:21:19.027 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=grafanaStorageLogger t=2024-09-17T12:21:18.649356349Z level=info msg="storage starting" 2024-09-17T12:21:19.027 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=ngalert.state.manager t=2024-09-17T12:21:18.64936569Z level=info msg="Warming state cache for startup" 2024-09-17T12:21:19.027 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=http.server t=2024-09-17T12:21:18.651178049Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-17T12:21:19.027 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=sqlstore.transactions t=2024-09-17T12:21:18.659678283Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T12:21:19.027 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=ngalert.state.manager t=2024-09-17T12:21:18.661898831Z level=info msg="State cache has been initialized" states=0 duration=12.531376ms 2024-09-17T12:21:19.027 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=ticker t=2024-09-17T12:21:18.661963674Z level=info msg=starting first_tick=2024-09-17T12:21:20Z 2024-09-17T12:21:19.027 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:21:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=ngalert.multiorg.alertmanager t=2024-09-17T12:21:18.661959234Z level=info msg="starting MultiOrg Alertmanager" 2024-09-17T12:21:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:19 smithi196 ceph-mon[27428]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:19.595 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:19 smithi016 ceph-mon[23719]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:19.596 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:19 smithi016 ceph-mon[34212]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:19.622 INFO:teuthology.orchestra.run.smithi016.stdout:133143986202 2024-09-17T12:21:19.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-17T12:21:20.863 INFO:teuthology.orchestra.run.smithi016.stdout:77309411365 2024-09-17T12:21:20.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-17T12:21:21.322 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:21.333 INFO:teuthology.orchestra.run.smithi016.stdout:240518168589 2024-09-17T12:21:21.333 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-17T12:21:21.560 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:21 smithi016 ceph-mon[23719]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:21.560 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:21 smithi016 ceph-mon[34212]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:21.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:21 smithi196 ceph-mon[27428]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:22.624 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:22 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:21:22.624 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:22 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:22.624 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:22 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:21:22.624 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:22 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:22.625 INFO:teuthology.orchestra.run.smithi016.stdout:55834574891 2024-09-17T12:21:22.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-17T12:21:22.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:22 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:21:22.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:22 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:23.485 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:23 smithi016 ceph-mon[23719]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:23.485 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:23 smithi016 ceph-mon[34212]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:23.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:23 smithi196 ceph-mon[27428]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:24.194 INFO:teuthology.orchestra.run.smithi016.stdout:34359738417 2024-09-17T12:21:24.194 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-17T12:21:25.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:25 smithi016 ceph-mon[23719]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:25.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:25 smithi016 ceph-mon[34212]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:25.578 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:25.579 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:25.604 INFO:teuthology.orchestra.run.smithi016.stdout:206158430228 2024-09-17T12:21:25.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-17T12:21:25.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:25 smithi196 ceph-mon[27428]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:26.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:21:26] "GET /metrics HTTP/1.1" 200 26196 "" "Prometheus/2.43.0" 2024-09-17T12:21:27.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:27 smithi016 ceph-mon[23719]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:27.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:27 smithi016 ceph-mon[34212]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:27.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:27 smithi196 ceph-mon[27428]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:29.630 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:29 smithi016 ceph-mon[23719]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:29.630 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:29 smithi016 ceph-mon[34212]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:29.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:29 smithi196 ceph-mon[27428]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:111669149730 2024-09-17T12:21:30.636 INFO:teuthology.orchestra.run.smithi016.stdout:133143986205 2024-09-17T12:21:31.351 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149727 got 111669149730 for osd.3 2024-09-17T12:21:31.351 DEBUG:teuthology.parallel:result is None 2024-09-17T12:21:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:31 smithi016 ceph-mon[23719]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:31 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/293915212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T12:21:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:31 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1214235461' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T12:21:31.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:31 smithi016 ceph-mon[34212]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:31.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:31 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/293915212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T12:21:31.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:31 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1214235461' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T12:21:31.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:31 smithi196 ceph-mon[27428]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:31.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:31 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/293915212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T12:21:31.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:31 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1214235461' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T12:21:32.165 INFO:teuthology.orchestra.run.smithi016.stdout:240518168592 2024-09-17T12:21:32.377 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:32 smithi016 bash[66038]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-17T12:21:32.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:32 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3662765756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T12:21:32.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:32 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3662765756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T12:21:32.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:32 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3662765756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T12:21:33.242 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:33.242 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:33.617 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:33 smithi016 ceph-mon[34212]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:33.617 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:33 smithi016 ceph-mon[23719]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:33.678 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:33 smithi196 ceph-mon[27428]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:33.900 INFO:teuthology.orchestra.run.smithi016.stdout:167503724569 2024-09-17T12:21:34.191 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:33 smithi016 bash[66038]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-17T12:21:34.603 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:34 smithi016 bash[66038]: Writing manifest to image destination 2024-09-17T12:21:34.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:34 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/604391168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T12:21:34.603 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:34 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/604391168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T12:21:34.624 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986202 got 133143986205 for osd.4 2024-09-17T12:21:34.624 DEBUG:teuthology.parallel:result is None 2024-09-17T12:21:34.695 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:34 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/604391168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T12:21:34.941 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:34 smithi016 podman[66038]: 2024-09-17 12:21:34.661369789 +0000 UTC m=+2.876257507 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-17T12:21:35.300 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724565 got 167503724569 for osd.5 2024-09-17T12:21:35.301 DEBUG:teuthology.parallel:result is None 2024-09-17T12:21:35.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-mon[23719]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:35.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-mon[34212]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:35.611 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 podman[66038]: 2024-09-17 12:21:35.489194577 +0000 UTC m=+3.704082282 container create c74c1d1d2736c5cde0ecf5ebda26ee74b22c314e12eff4fb062410a9374ea072 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:35.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:35 smithi196 ceph-mon[27428]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:36.191 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 podman[66038]: 2024-09-17 12:21:35.918954741 +0000 UTC m=+4.133842460 container init c74c1d1d2736c5cde0ecf5ebda26ee74b22c314e12eff4fb062410a9374ea072 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:36.192 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 podman[66038]: 2024-09-17 12:21:35.92239829 +0000 UTC m=+4.137285990 container start c74c1d1d2736c5cde0ecf5ebda26ee74b22c314e12eff4fb062410a9374ea072 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:36.192 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.926Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-17T12:21:36.192 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.926Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-17T12:21:36.192 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-17T12:21:36.192 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T12:21:36.192 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-17T12:21:36.192 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-17T12:21:36.192 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-17T12:21:36.192 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=arp 2024-09-17T12:21:36.192 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-17T12:21:36.192 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-17T12:21:36.193 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-17T12:21:36.193 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-17T12:21:36.193 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-17T12:21:36.193 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-17T12:21:36.193 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-17T12:21:36.193 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-17T12:21:36.193 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=edac 2024-09-17T12:21:36.194 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-17T12:21:36.194 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-17T12:21:36.194 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-17T12:21:36.194 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-17T12:21:36.194 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-17T12:21:36.194 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-17T12:21:36.194 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-17T12:21:36.194 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-17T12:21:36.194 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-17T12:21:36.194 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-17T12:21:36.194 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.927Z caller=node_exporter.go:117 level=info collector=os 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-17T12:21:36.195 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=stat 2024-09-17T12:21:36.196 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-17T12:21:36.196 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-17T12:21:36.196 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-17T12:21:36.196 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=time 2024-09-17T12:21:36.196 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-17T12:21:36.196 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=uname 2024-09-17T12:21:36.196 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-17T12:21:36.196 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-17T12:21:36.196 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-17T12:21:36.196 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-17T12:21:36.196 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a[66276]: ts=2024-09-17T12:21:35.928Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-17T12:21:36.399 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168589 got 240518168592 for osd.7 2024-09-17T12:21:36.399 DEBUG:teuthology.parallel:result is None 2024-09-17T12:21:36.493 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:21:36] "GET /metrics HTTP/1.1" 200 26196 "" "Prometheus/2.43.0" 2024-09-17T12:21:37.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:37 smithi016 ceph-mon[23719]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:37.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:37 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:21:37.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:37 smithi016 ceph-mon[34212]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:37.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:37 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:21:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:37 smithi196 ceph-mon[27428]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:37 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:21:38.191 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:38 smithi016 bash[66038]: c74c1d1d2736c5cde0ecf5ebda26ee74b22c314e12eff4fb062410a9374ea072 2024-09-17T12:21:38.191 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:21:38 smithi016 systemd[1]: Started Ceph node-exporter.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:21:38.229 INFO:teuthology.orchestra.run.smithi016.stdout:77309411368 2024-09-17T12:21:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:55834574894 2024-09-17T12:21:39.197 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:39 smithi196 bash[56587]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-17T12:21:39.268 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411365 got 77309411368 for osd.2 2024-09-17T12:21:39.268 DEBUG:teuthology.parallel:result is None 2024-09-17T12:21:39.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:39 smithi196 ceph-mon[27428]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:39.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:39 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:39.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:39 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:39.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:39 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:39.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:39 smithi196 ceph-mon[27428]: Deploying daemon node-exporter.b on smithi196 2024-09-17T12:21:39.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:39 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2284619321' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T12:21:39.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:39 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/55800769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T12:21:39.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[23719]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:39.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:39.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:39.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:39.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[23719]: Deploying daemon node-exporter.b on smithi196 2024-09-17T12:21:39.567 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2284619321' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T12:21:39.567 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/55800769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T12:21:39.567 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[34212]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:39.567 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:39.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:39.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:39.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[34212]: Deploying daemon node-exporter.b on smithi196 2024-09-17T12:21:39.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2284619321' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T12:21:39.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:39 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/55800769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T12:21:39.718 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574891 got 55834574894 for osd.1 2024-09-17T12:21:39.718 DEBUG:teuthology.parallel:result is None 2024-09-17T12:21:39.776 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:39 smithi196 bash[56587]: Getting image source signatures 2024-09-17T12:21:39.776 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:39 smithi196 bash[56587]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-17T12:21:39.776 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:39 smithi196 bash[56587]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-17T12:21:39.777 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:39 smithi196 bash[56587]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-17T12:21:41.199 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:40 smithi196 bash[56587]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-17T12:21:41.199 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 bash[56587]: Writing manifest to image destination 2024-09-17T12:21:41.271 INFO:teuthology.orchestra.run.smithi016.stdout:206158430231 2024-09-17T12:21:41.463 INFO:teuthology.orchestra.run.smithi016.stdout:34359738420 2024-09-17T12:21:41.464 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:41 smithi016 ceph-mon[23719]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:41.464 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:41 smithi016 ceph-mon[34212]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:41.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-mon[27428]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:41.473 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 podman[56587]: 2024-09-17 12:21:41.260327146 +0000 UTC m=+2.257760146 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-17T12:21:41.474 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 podman[56587]: 2024-09-17 12:21:41.375367231 +0000 UTC m=+2.372800219 container create eaddaabd45094145f61185108b219cf959aacd825d7294de24714d8f0c40e29f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T12:21:42.027 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 podman[56587]: 2024-09-17 12:21:41.560688049 +0000 UTC m=+2.558121044 container init eaddaabd45094145f61185108b219cf959aacd825d7294de24714d8f0c40e29f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T12:21:42.027 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 podman[56587]: 2024-09-17 12:21:41.563598204 +0000 UTC m=+2.561031192 container start eaddaabd45094145f61185108b219cf959aacd825d7294de24714d8f0c40e29f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T12:21:42.027 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.567Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-17T12:21:42.027 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.567Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-17T12:21:42.027 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.567Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-17T12:21:42.027 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.567Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T12:21:42.027 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-17T12:21:42.027 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-17T12:21:42.027 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-17T12:21:42.027 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=arp 2024-09-17T12:21:42.027 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-17T12:21:42.027 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-17T12:21:42.028 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-17T12:21:42.028 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-17T12:21:42.028 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-17T12:21:42.028 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-17T12:21:42.028 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-17T12:21:42.028 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-17T12:21:42.028 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=edac 2024-09-17T12:21:42.029 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-17T12:21:42.029 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-17T12:21:42.029 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-17T12:21:42.029 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-17T12:21:42.029 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-17T12:21:42.029 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-17T12:21:42.029 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-17T12:21:42.029 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-17T12:21:42.029 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-17T12:21:42.029 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-17T12:21:42.029 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-17T12:21:42.029 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-17T12:21:42.029 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.568Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=os 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=stat 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-17T12:21:42.030 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-17T12:21:42.031 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-17T12:21:42.031 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=time 2024-09-17T12:21:42.031 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-17T12:21:42.031 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=uname 2024-09-17T12:21:42.031 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-17T12:21:42.031 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-17T12:21:42.031 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-17T12:21:42.031 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-17T12:21:42.031 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b[56662]: ts=2024-09-17T12:21:41.569Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-17T12:21:42.031 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 bash[56587]: eaddaabd45094145f61185108b219cf959aacd825d7294de24714d8f0c40e29f 2024-09-17T12:21:42.032 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:21:41 smithi196 systemd[1]: Started Ceph node-exporter.b for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:21:42.122 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430228 got 206158430231 for osd.6 2024-09-17T12:21:42.122 DEBUG:teuthology.parallel:result is None 2024-09-17T12:21:42.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2684220533' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T12:21:42.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3154125159' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T12:21:42.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:21:42.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2684220533' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T12:21:42.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3154125159' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T12:21:42.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:42 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:21:42.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:42 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2684220533' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T12:21:42.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:42 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3154125159' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T12:21:42.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:42 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:42 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:42 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:42 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:42 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:42 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:42.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:42 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:21:42.702 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738420 for osd.0 2024-09-17T12:21:42.703 DEBUG:teuthology.parallel:result is None 2024-09-17T12:21:42.703 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-17T12:21:42.703 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T12:21:43.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:43 smithi016 ceph-mon[23719]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:43.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:43 smithi016 ceph-mon[34212]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:43.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:43 smithi196 ceph-mon[27428]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:43.511 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:45.080 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:21:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-17T12:21:45.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:45 smithi016 ceph-mon[23719]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:45.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:45 smithi016 ceph-mon[34212]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:45.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:45 smithi196 ceph-mon[27428]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:46.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:46 smithi016 ceph-mon[23719]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T12:21:46.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:46 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:46.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:46 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:46.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:21:46] "GET /metrics HTTP/1.1" 200 26203 "" "Prometheus/2.43.0" 2024-09-17T12:21:46.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:46 smithi016 ceph-mon[34212]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T12:21:46.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:46 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:46.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:46 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:46.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:46 smithi196 ceph-mon[27428]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T12:21:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:46 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:46 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:46.858 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-09-17T12:21:43.695424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":230824,"kb_used_data":6476,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":749730392,"statfs":{"total":767960285184,"available":767723921408,"internally_reserved":0,"allocated":6631424,"data_stored":3740736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229625401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002080"},"pg_stats":[{"pgid":"1.0","version":"61'128","reported_seq":258,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-17T12:20:53.741876+0000","last_change":"2024-09-17T12:20:25.291733+0000","last_active":"2024-09-17T12:20:53.741876+0000","last_peered":"2024-09-17T12:20:53.741876+0000","last_clean":"2024-09-17T12:20:53.741876+0000","last_became_active":"2024-09-17T12:20:25.291357+0000","last_became_peered":"2024-09-17T12:20:25.291357+0000","last_unstale":"2024-09-17T12:20:53.741876+0000","last_undegraded":"2024-09-17T12:20:53.741876+0000","last_fullsized":"2024-09-17T12:20:53.741876+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T12:18:25.358269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T12:18:25.358269+0000","last_clean_scrub_stamp":"2024-09-17T12:18:25.358269+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:55:39.404820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168595,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28192,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716960,"statfs":{"total":95995035648,"available":95966167040,"internally_reserved":0,"allocated":1200128,"data_stored":836572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.71799999999999997}]},{"osd":1,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.64600000000000002}]},{"osd":2,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.75}]},{"osd":3,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.67800000000000005}]},{"osd":4,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.52900000000000003}]},{"osd":5,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.58399999999999996}]},{"osd":6,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57199999999999995}]}]},{"osd":6,"up_from":48,"seq":206158430232,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1200128,"data_stored":836572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.36899999999999999}]},{"osd":1,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.39500000000000002}]},{"osd":2,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.47199999999999998}]},{"osd":3,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.45100000000000001}]},{"osd":4,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.379},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.33300000000000002}]},{"osd":5,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.29499999999999998}]},{"osd":7,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.435}]}]},{"osd":5,"up_from":39,"seq":167503724572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717540,"statfs":{"total":95995035648,"available":95966760960,"internally_reserved":0,"allocated":606208,"data_stored":246204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73499999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54200000000000004,"15min":0.54500000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.71899999999999997,"5min":0.755,"15min":0.755},"last":0.317}]},{"osd":1,"last update":"Tue Sep 17 12:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.375},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67300000000000004,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.35099999999999998}]},{"osd":2,"last update":"Tue Sep 17 12:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57799999999999996,"15min":0.58299999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73699999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.442}]},{"osd":3,"last update":"Tue Sep 17 12:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.65400000000000003,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57099999999999995,"15min":0.57199999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77400000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.40999999999999998}]},{"osd":4,"last update":"Tue Sep 17 12:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.39000000000000001,"15min":0.378},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.45400000000000001}]},{"osd":6,"last update":"Tue Sep 17 12:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.38900000000000001}]},{"osd":7,"last update":"Tue Sep 17 12:21:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.47099999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986208,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717540,"statfs":{"total":95995035648,"available":95966760960,"internally_reserved":0,"allocated":606208,"data_stored":246204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.441,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72999999999999998,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.45800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73899999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.51400000000000001}]},{"osd":1,"last update":"Tue Sep 17 12:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.50800000000000001,"15min":0.50900000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.38},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.58599999999999997,"15min":0.55400000000000005},"min":{"1min":0.38400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":6.9500000000000002,"5min":6.9500000000000002,"15min":6.9500000000000002},"last":0.52300000000000002}]},{"osd":2,"last update":"Tue Sep 17 12:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51200000000000001,"15min":0.51300000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68899999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68000000000000005,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.39000000000000001}]},{"osd":3,"last update":"Tue Sep 17 12:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54100000000000004,"15min":0.54400000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.66300000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.40799999999999997}]},{"osd":5,"last update":"Tue Sep 17 12:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.36399999999999999,"15min":0.36099999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.443,"5min":0.40000000000000002,"15min":0.39300000000000002},"min":{"1min":0.30299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.41899999999999998}]},{"osd":6,"last update":"Tue Sep 17 12:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.502}]},{"osd":7,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.435}]}]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717540,"statfs":{"total":95995035648,"available":95966760960,"internally_reserved":0,"allocated":606208,"data_stored":246204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:20:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41099999999999998,"15min":0.41199999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.52700000000000002,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.45500000000000002,"15min":0.45200000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.30499999999999999}]},{"osd":1,"last update":"Tue Sep 17 12:20:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.47499999999999998,"15min":0.47899999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.34499999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.60299999999999998,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.36599999999999999}]},{"osd":2,"last update":"Tue Sep 17 12:20:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.435},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.46000000000000002,"15min":0.45000000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.38700000000000001}]},{"osd":4,"last update":"Tue Sep 17 12:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58699999999999997,"15min":0.58599999999999997},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.45100000000000001}]},{"osd":5,"last update":"Tue Sep 17 12:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.61099999999999999,"15min":0.61299999999999999},"min":{"1min":0.443,"5min":0.437,"15min":0.437},"max":{"1min":0.80900000000000005,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61599999999999999,"15min":0.61899999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.5}]},{"osd":6,"last update":"Tue Sep 17 12:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42399999999999999}]},{"osd":7,"last update":"Tue Sep 17 12:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.63900000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31708,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713444,"statfs":{"total":95995035648,"available":95962566656,"internally_reserved":0,"allocated":606208,"data_stored":246204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:21:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.35899999999999999,"15min":0.34899999999999998},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.53100000000000003,"5min":0.626,"15min":0.626},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.38300000000000001,"15min":0.36499999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.58299999999999996,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.44}]},{"osd":1,"last update":"Tue Sep 17 12:21:29 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38700000000000001,"15min":0.378},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52800000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.41499999999999998,"15min":0.40400000000000003},"min":{"1min":0.29599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.50700000000000001}]},{"osd":3,"last update":"Tue Sep 17 12:20:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.46600000000000003,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.58099999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.41799999999999998}]},{"osd":4,"last update":"Tue Sep 17 12:21:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57099999999999995,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83599999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60499999999999998,"15min":0.60999999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.75700000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.66600000000000004}]},{"osd":5,"last update":"Tue Sep 17 12:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57899999999999996,"15min":0.57999999999999996},"min":{"1min":0.434,"5min":0.433,"15min":0.433},"max":{"1min":0.80300000000000005,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.62},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58699999999999997,"15min":0.58999999999999997},"min":{"1min":0.40799999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73799999999999999,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.65000000000000002}]},{"osd":6,"last update":"Tue Sep 17 12:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.63800000000000001}]},{"osd":7,"last update":"Tue Sep 17 12:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.69599999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717540,"statfs":{"total":95995035648,"available":95966760960,"internally_reserved":0,"allocated":606208,"data_stored":246204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.33300000000000002,"15min":0.30099999999999999},"min":{"1min":0.27000000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.36099999999999999,"15min":0.32900000000000001},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.58299999999999996}]},{"osd":2,"last update":"Tue Sep 17 12:21:32 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.41299999999999998,"15min":0.40600000000000003},"min":{"1min":0.26200000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.39300000000000002,"15min":0.38100000000000001},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.56000000000000005}]},{"osd":3,"last update":"Tue Sep 17 12:20:48 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48499999999999999,"15min":0.48099999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.42799999999999999,"15min":0.42299999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.59999999999999998}]},{"osd":4,"last update":"Tue Sep 17 12:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.628,"15min":0.627},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.75900000000000001}]},{"osd":5,"last update":"Tue Sep 17 12:21:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40000000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.745},{"interface":"front","average":{"1min":0.626,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.77500000000000002}]},{"osd":6,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.80200000000000005}]},{"osd":7,"last update":"Tue Sep 17 12:21:24 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.82199999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1200128,"data_stored":836572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 12:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.33700000000000002,"15min":0.32400000000000001},"min":{"1min":0.22700000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.378,"15min":0.35399999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.35899999999999999}]},{"osd":2,"last update":"Tue Sep 17 12:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.39300000000000002,"15min":0.38},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38100000000000001,"15min":0.371},"min":{"1min":0.28199999999999997,"5min":0.245,"15min":0.245},"max":{"1min":0.56299999999999994,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.57199999999999995}]},{"osd":3,"last update":"Tue Sep 17 12:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.46400000000000002,"15min":0.45800000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44,"15min":0.438},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.66600000000000004}]},{"osd":4,"last update":"Tue Sep 17 12:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79300000000000004,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.38700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80600000000000005,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.51200000000000001}]},{"osd":5,"last update":"Tue Sep 17 12:21:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58999999999999997,"15min":0.59099999999999997},"min":{"1min":0.45400000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82499999999999996,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.83499999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.49199999999999999}]},{"osd":6,"last update":"Tue Sep 17 12:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.67800000000000005}]},{"osd":7,"last update":"Tue Sep 17 12:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.624},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.70399999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T12:21:46.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T12:21:47.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:47 smithi016 ceph-mon[23719]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:47.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:47 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:47.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:47 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:47.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:47 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:21:47.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:47 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:21:47.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:47 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:47.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:47 smithi016 ceph-mon[34212]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:47.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:47 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:47.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:47 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:47.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:47 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:21:47.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:47 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:21:47.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:47 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:47.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:47 smithi196 ceph-mon[27428]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:47.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:47 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:47.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:47 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:47.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:47 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:21:47.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:47 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:21:47.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:47 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:48.477 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:48 smithi016 ceph-mon[23719]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T12:21:48.477 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:48 smithi016 ceph-mon[23719]: Reconfiguring daemon alertmanager.a on smithi016 2024-09-17T12:21:48.477 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:48 smithi016 ceph-mon[34212]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T12:21:48.477 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:48 smithi016 ceph-mon[34212]: Reconfiguring daemon alertmanager.a on smithi016 2024-09-17T12:21:48.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:48 smithi196 ceph-mon[27428]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T12:21:48.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:48 smithi196 ceph-mon[27428]: Reconfiguring daemon alertmanager.a on smithi016 2024-09-17T12:21:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:49.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:49 smithi016 ceph-mon[23719]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:49.500 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:49 smithi016 ceph-mon[34212]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:49.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:49 smithi196 ceph-mon[27428]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:49.784 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:49 smithi016 systemd[1]: Stopping Ceph alertmanager.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:21:51.312 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:51 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[63742]: ts=2024-09-17T12:21:51.045Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-17T12:21:51.313 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:51 smithi016 podman[67517]: 2024-09-17 12:21:51.214905197 +0000 UTC m=+0.463501693 container died 83f08a029887f9669dca2893a4dea1b604447a544afa224542ca1f0d5b8abd82 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:51.398 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:21:51.399 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-17T12:21:51.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:51 smithi016 ceph-mon[34212]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:51.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:51 smithi016 ceph-mon[23719]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:51.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:51 smithi196 ceph-mon[27428]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:51.941 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:51 smithi016 podman[67517]: 2024-09-17 12:21:51.781702209 +0000 UTC m=+1.030298704 container cleanup 83f08a029887f9669dca2893a4dea1b604447a544afa224542ca1f0d5b8abd82 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:51.941 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:51 smithi016 bash[67517]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a 2024-09-17T12:21:52.291 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:51 smithi016 podman[67535]: 2024-09-17 12:21:51.999553931 +0000 UTC m=+0.911144145 container remove 83f08a029887f9669dca2893a4dea1b604447a544afa224542ca1f0d5b8abd82 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:52.291 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:52 smithi016 podman[67535]: 2024-09-17 12:21:52.050263559 +0000 UTC m=+0.961853772 volume remove 7ad842b87448fa353a13cfa220db0c870b02d0a0904a89decb4cd866c8e5d8f1 2024-09-17T12:21:52.434 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-09-17T12:21:49.696360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":230824,"kb_used_data":6476,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":749730392,"statfs":{"total":767960285184,"available":767723921408,"internally_reserved":0,"allocated":6631424,"data_stored":3740736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229625401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001953"},"pg_stats":[{"pgid":"1.0","version":"61'128","reported_seq":258,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-17T12:20:53.741876+0000","last_change":"2024-09-17T12:20:25.291733+0000","last_active":"2024-09-17T12:20:53.741876+0000","last_peered":"2024-09-17T12:20:53.741876+0000","last_clean":"2024-09-17T12:20:53.741876+0000","last_became_active":"2024-09-17T12:20:25.291357+0000","last_became_peered":"2024-09-17T12:20:25.291357+0000","last_unstale":"2024-09-17T12:20:53.741876+0000","last_undegraded":"2024-09-17T12:20:53.741876+0000","last_fullsized":"2024-09-17T12:20:53.741876+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T12:18:25.358269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T12:18:25.358269+0000","last_clean_scrub_stamp":"2024-09-17T12:18:25.358269+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:55:39.404820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168596,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28192,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716960,"statfs":{"total":95995035648,"available":95966167040,"internally_reserved":0,"allocated":1200128,"data_stored":836572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.49199999999999999}]},{"osd":1,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.498},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.46600000000000003}]},{"osd":2,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.56100000000000005}]},{"osd":3,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.442}]},{"osd":4,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.30299999999999999}]},{"osd":5,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.36599999999999999}]},{"osd":6,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.35399999999999998}]}]},{"osd":6,"up_from":48,"seq":206158430233,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1200128,"data_stored":836572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.54800000000000004}]},{"osd":1,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.45500000000000002}]},{"osd":2,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.42799999999999999}]},{"osd":3,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.53800000000000003}]},{"osd":4,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.60199999999999998}]},{"osd":5,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.64000000000000001}]},{"osd":7,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.58899999999999997}]}]},{"osd":5,"up_from":39,"seq":167503724573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717540,"statfs":{"total":95995035648,"available":95966760960,"internally_reserved":0,"allocated":606208,"data_stored":246204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73499999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54200000000000004,"15min":0.54500000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.71899999999999997,"5min":0.755,"15min":0.755},"last":0.377}]},{"osd":1,"last update":"Tue Sep 17 12:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67300000000000004,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.44700000000000001}]},{"osd":2,"last update":"Tue Sep 17 12:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57799999999999996,"15min":0.58299999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73699999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.41499999999999998}]},{"osd":3,"last update":"Tue Sep 17 12:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.65400000000000003,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57099999999999995,"15min":0.57199999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77400000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.46000000000000002}]},{"osd":4,"last update":"Tue Sep 17 12:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.39000000000000001,"15min":0.378},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.433}]},{"osd":6,"last update":"Tue Sep 17 12:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.53800000000000003}]},{"osd":7,"last update":"Tue Sep 17 12:21:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.502}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717540,"statfs":{"total":95995035648,"available":95966760960,"internally_reserved":0,"allocated":606208,"data_stored":246204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.441,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72999999999999998,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.45800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73899999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.56100000000000005}]},{"osd":1,"last update":"Tue Sep 17 12:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.50800000000000001,"15min":0.50900000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.58599999999999997,"15min":0.55400000000000005},"min":{"1min":0.38400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":6.9500000000000002,"5min":6.9500000000000002,"15min":6.9500000000000002},"last":0.57499999999999996}]},{"osd":2,"last update":"Tue Sep 17 12:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51200000000000001,"15min":0.51300000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68899999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68000000000000005,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.50800000000000001}]},{"osd":3,"last update":"Tue Sep 17 12:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54100000000000004,"15min":0.54400000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.439},{"interface":"front","average":{"1min":0.499,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.66300000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.54900000000000004}]},{"osd":5,"last update":"Tue Sep 17 12:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.36399999999999999,"15min":0.36099999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.40000000000000002,"15min":0.39300000000000002},"min":{"1min":0.30299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.34100000000000003}]},{"osd":6,"last update":"Tue Sep 17 12:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.38500000000000001}]},{"osd":7,"last update":"Tue Sep 17 12:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.309},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.32500000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717540,"statfs":{"total":95995035648,"available":95966760960,"internally_reserved":0,"allocated":606208,"data_stored":246204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:20:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41099999999999998,"15min":0.41199999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.52700000000000002,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.45500000000000002,"15min":0.45200000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.29399999999999998}]},{"osd":1,"last update":"Tue Sep 17 12:20:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.47499999999999998,"15min":0.47899999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.34499999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.60299999999999998,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.26100000000000001}]},{"osd":2,"last update":"Tue Sep 17 12:20:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.46000000000000002,"15min":0.45000000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.27500000000000002}]},{"osd":4,"last update":"Tue Sep 17 12:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58699999999999997,"15min":0.58599999999999997},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.59099999999999997}]},{"osd":5,"last update":"Tue Sep 17 12:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.61099999999999999,"15min":0.61299999999999999},"min":{"1min":0.443,"5min":0.437,"15min":0.437},"max":{"1min":0.80900000000000005,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61599999999999999,"15min":0.61899999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.63900000000000001}]},{"osd":6,"last update":"Tue Sep 17 12:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.65200000000000002}]},{"osd":7,"last update":"Tue Sep 17 12:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.623}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31708,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713444,"statfs":{"total":95995035648,"available":95962566656,"internally_reserved":0,"allocated":606208,"data_stored":246204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:21:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.35899999999999999,"15min":0.34899999999999998},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.53100000000000003,"5min":0.626,"15min":0.626},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.38300000000000001,"15min":0.36499999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.58299999999999996,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.39900000000000002}]},{"osd":1,"last update":"Tue Sep 17 12:21:29 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38700000000000001,"15min":0.378},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52800000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.41499999999999998,"15min":0.40400000000000003},"min":{"1min":0.29599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.46800000000000003}]},{"osd":3,"last update":"Tue Sep 17 12:20:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.46600000000000003,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.58099999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.45700000000000002}]},{"osd":4,"last update":"Tue Sep 17 12:21:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57099999999999995,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83599999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60499999999999998,"15min":0.60999999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.75700000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.61799999999999999}]},{"osd":5,"last update":"Tue Sep 17 12:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57899999999999996,"15min":0.57999999999999996},"min":{"1min":0.434,"5min":0.433,"15min":0.433},"max":{"1min":0.80300000000000005,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.505},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58699999999999997,"15min":0.58999999999999997},"min":{"1min":0.40799999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73799999999999999,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60199999999999998}]},{"osd":6,"last update":"Tue Sep 17 12:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.65400000000000003}]},{"osd":7,"last update":"Tue Sep 17 12:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.68500000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717540,"statfs":{"total":95995035648,"available":95966760960,"internally_reserved":0,"allocated":606208,"data_stored":246204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 12:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.33300000000000002,"15min":0.30099999999999999},"min":{"1min":0.27000000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.441,"5min":0.36099999999999999,"15min":0.32900000000000001},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.48099999999999998}]},{"osd":2,"last update":"Tue Sep 17 12:21:32 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.41299999999999998,"15min":0.40600000000000003},"min":{"1min":0.26200000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.39300000000000002,"15min":0.38100000000000001},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.45700000000000002}]},{"osd":3,"last update":"Tue Sep 17 12:20:48 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48499999999999999,"15min":0.48099999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.42799999999999999,"15min":0.42299999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.55000000000000004}]},{"osd":4,"last update":"Tue Sep 17 12:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.628,"15min":0.627},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53100000000000003}]},{"osd":5,"last update":"Tue Sep 17 12:21:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40000000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.622}]},{"osd":6,"last update":"Tue Sep 17 12:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.59999999999999998}]},{"osd":7,"last update":"Tue Sep 17 12:21:24 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.58299999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1200128,"data_stored":836572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 12:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.33700000000000002,"15min":0.32400000000000001},"min":{"1min":0.22700000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.378,"15min":0.35399999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.25900000000000001}]},{"osd":2,"last update":"Tue Sep 17 12:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.39300000000000002,"15min":0.38},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38100000000000001,"15min":0.371},"min":{"1min":0.28199999999999997,"5min":0.245,"15min":0.245},"max":{"1min":0.56299999999999994,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.24299999999999999}]},{"osd":3,"last update":"Tue Sep 17 12:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.46400000000000002,"15min":0.45800000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44,"15min":0.438},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.27900000000000003}]},{"osd":4,"last update":"Tue Sep 17 12:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79300000000000004,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.44},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.38700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80600000000000005,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.40699999999999997}]},{"osd":5,"last update":"Tue Sep 17 12:21:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58999999999999997,"15min":0.59099999999999997},"min":{"1min":0.45400000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82499999999999996,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.83499999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.46100000000000002}]},{"osd":6,"last update":"Tue Sep 17 12:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.376}]},{"osd":7,"last update":"Tue Sep 17 12:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.47499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T12:21:52.436 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-17T12:21:52.436 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T12:21:52.437 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-17T12:21:52.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-17T12:21:52.570 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:52 smithi016 ceph-mon[23719]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T12:21:52.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:21:52.571 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:52 smithi016 ceph-mon[34212]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T12:21:52.571 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:21:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:52.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:52 smithi196 ceph-mon[27428]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T12:21:52.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:21:52.899 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:52 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T12:21:52.899 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:52 smithi016 systemd[1]: Stopped Ceph alertmanager.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:21:52.899 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:52 smithi016 systemd[1]: Starting Ceph alertmanager.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:21:52.899 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:52 smithi016 podman[67729]: 2024-09-17 12:21:52.714056222 +0000 UTC m=+0.022237000 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-17T12:21:53.180 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:52 smithi016 podman[67729]: 2024-09-17 12:21:52.981050379 +0000 UTC m=+0.289231144 volume create 3931f1a1af23e69e4d642186ff9d0dbf793b4d3fd0ee2f79f2c4fce7cfaedc47 2024-09-17T12:21:53.180 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:53 smithi016 podman[67729]: 2024-09-17 12:21:53.056335924 +0000 UTC m=+0.364516688 container create 5103666cca2ef272a7710277faa79faf0a82281aaf1271a9a34842aefcf1eff0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:53 smithi016 ceph-mon[23719]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:53.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:53 smithi016 ceph-mon[34212]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:53.442 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:53 smithi016 podman[67729]: 2024-09-17 12:21:53.259015414 +0000 UTC m=+0.567196178 container init 5103666cca2ef272a7710277faa79faf0a82281aaf1271a9a34842aefcf1eff0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:53.442 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:53 smithi016 podman[67729]: 2024-09-17 12:21:53.2619561 +0000 UTC m=+0.570136865 container start 5103666cca2ef272a7710277faa79faf0a82281aaf1271a9a34842aefcf1eff0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:53.442 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:21:53.275Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-17T12:21:53.442 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:21:53.275Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-17T12:21:53.442 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:21:53.276Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.16 port=9094 2024-09-17T12:21:53.442 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:21:53.277Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T12:21:53.442 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:21:53.300Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T12:21:53.442 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:21:53.301Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T12:21:53.442 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:21:53.304Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-17T12:21:53.442 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:21:53.304Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-17T12:21:53.442 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:53 smithi016 bash[67729]: 5103666cca2ef272a7710277faa79faf0a82281aaf1271a9a34842aefcf1eff0 2024-09-17T12:21:53.443 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:53 smithi016 systemd[1]: Started Ceph alertmanager.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:21:53.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:53 smithi196 ceph-mon[27428]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:54.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:54.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:54.646 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:54 smithi016 ceph-mon[23719]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T12:21:54.646 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:54.646 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:54.646 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:54 smithi016 ceph-mon[34212]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T12:21:54.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:54.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:54.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:54 smithi196 ceph-mon[27428]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T12:21:55.033 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-17T12:21:55.033 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T12:21:55.322 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:21:55 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:21:55.277Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000616382s 2024-09-17T12:21:55.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:55 smithi016 ceph-mon[23719]: Reconfiguring daemon prometheus.a on smithi196 2024-09-17T12:21:55.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:55 smithi016 ceph-mon[23719]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:55.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:55 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2801745661' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T12:21:55.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:55 smithi016 ceph-mon[34212]: Reconfiguring daemon prometheus.a on smithi196 2024-09-17T12:21:55.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:55 smithi016 ceph-mon[34212]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:55.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:55 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2801745661' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T12:21:55.704 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:55 smithi196 ceph-mon[27428]: Reconfiguring daemon prometheus.a on smithi196 2024-09-17T12:21:55.704 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:55 smithi196 ceph-mon[27428]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:55.704 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:55 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2801745661' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T12:21:55.723 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-17T12:21:55.723 INFO:tasks.cephadm:Setup complete, yielding 2024-09-17T12:21:55.724 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T12:21:55.734 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-09-17T12:21:55.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-09-17T12:21:55.957 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:21:56.368 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:21:56] "GET /metrics HTTP/1.1" 200 26201 "" "Prometheus/2.43.0" 2024-09-17T12:21:56.962 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 systemd[1]: Stopping Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:21:57.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[54362]: ts=2024-09-17T12:21:56.960Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T12:21:57.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[54362]: ts=2024-09-17T12:21:56.961Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-17T12:21:57.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[54362]: ts=2024-09-17T12:21:56.961Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-17T12:21:57.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[54362]: ts=2024-09-17T12:21:56.961Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T12:21:57.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[54362]: ts=2024-09-17T12:21:56.961Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-17T12:21:57.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[54362]: ts=2024-09-17T12:21:56.961Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-17T12:21:57.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[54362]: ts=2024-09-17T12:21:56.961Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T12:21:57.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[54362]: ts=2024-09-17T12:21:56.961Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-17T12:21:57.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[54362]: ts=2024-09-17T12:21:56.961Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-17T12:21:57.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[54362]: ts=2024-09-17T12:21:56.961Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T12:21:57.278 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[54362]: ts=2024-09-17T12:21:56.961Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-17T12:21:57.278 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[54362]: ts=2024-09-17T12:21:56.961Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-17T12:21:57.278 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:56 smithi196 podman[57416]: 2024-09-17 12:21:56.971074899 +0000 UTC m=+0.121276904 container died 1dc2f6a78d575d61ef8fe9242304e6ea4b4ebc9c6675a2dae200cbe6e0b67485 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:57.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:57 smithi016 ceph-mon[23719]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:57.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:57 smithi016 ceph-mon[34212]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:57.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:57 smithi196 ceph-mon[27428]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:57.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:57 smithi196 podman[57416]: 2024-09-17 12:21:57.460617235 +0000 UTC m=+0.610819232 container cleanup 1dc2f6a78d575d61ef8fe9242304e6ea4b4ebc9c6675a2dae200cbe6e0b67485 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:57.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:57 smithi196 bash[57416]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T12:21:58.027 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:57 smithi196 podman[57430]: 2024-09-17 12:21:57.728559147 +0000 UTC m=+0.754939968 container remove 1dc2f6a78d575d61ef8fe9242304e6ea4b4ebc9c6675a2dae200cbe6e0b67485 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:58.027 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:57 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T12:21:58.027 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:57 smithi196 systemd[1]: Stopped Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:21:58.027 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:57 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.075s CPU time. 2024-09-17T12:21:58.027 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:57 smithi196 systemd[1]: Starting Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:21:58.358 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 podman[57521]: 2024-09-17 12:21:58.008015882 +0000 UTC m=+0.016993454 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-17T12:21:58.358 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 podman[57521]: 2024-09-17 12:21:58.121118023 +0000 UTC m=+0.130095595 container create cb4f31b0d3509a34e9393e660fdd591b69131463e39da2fcf5a357f0ffadbb5e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:58.358 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 podman[57521]: 2024-09-17 12:21:58.323190614 +0000 UTC m=+0.332168186 container init cb4f31b0d3509a34e9393e660fdd591b69131463e39da2fcf5a357f0ffadbb5e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:58.358 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 podman[57521]: 2024-09-17 12:21:58.326028428 +0000 UTC m=+0.335006002 container start cb4f31b0d3509a34e9393e660fdd591b69131463e39da2fcf5a357f0ffadbb5e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:21:58.358 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.349Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-17T12:21:58.358 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.349Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-17T12:21:58.358 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.349Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi196 (none))" 2024-09-17T12:21:58.358 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.349Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T12:21:58.358 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.349Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T12:21:58.358 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.351Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-17T12:21:58.359 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.352Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-17T12:21:58.359 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.352Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-17T12:21:58.359 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.352Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-17T12:21:58.359 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.354Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-17T12:21:58.359 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.354Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.426µs 2024-09-17T12:21:58.359 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.354Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-17T12:21:58.359 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.356Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-17T12:21:58.359 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.356Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-17T12:21:58.359 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.356Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.832µs wal_replay_duration=2.566425ms wbl_replay_duration=144ns total_replay_duration=2.617575ms 2024-09-17T12:21:58.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.357Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-17T12:21:58.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.357Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-17T12:21:58.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.357Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T12:21:58.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.379Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=22.135745ms db_storage=738ns remote_storage=1.597µs web_handler=391ns query_engine=586ns scrape=378.502µs scrape_sd=88.341µs notify=9.532µs notify_sd=27.107µs rules=21.189627ms tracing=5.868µs 2024-09-17T12:21:58.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.380Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-17T12:21:58.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:21:58.380Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-17T12:21:58.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 bash[57521]: cb4f31b0d3509a34e9393e660fdd591b69131463e39da2fcf5a357f0ffadbb5e 2024-09-17T12:21:58.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:21:58 smithi196 systemd[1]: Started Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:21:58.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:58 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:58] ENGINE Bus STOPPING 2024-09-17T12:21:59.221 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:58 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T12:21:59.221 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:58 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:58] ENGINE Bus STOPPED 2024-09-17T12:21:59.221 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:58 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:58] ENGINE Bus STARTING 2024-09-17T12:21:59.221 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:59] ENGINE Serving on http://:::9283 2024-09-17T12:21:59.221 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:59] ENGINE Bus STARTED 2024-09-17T12:21:59.221 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:59] ENGINE Bus STOPPING 2024-09-17T12:21:59.429 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:21:59.429 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "d0fc053a-3d7f-40aa-976d-e3de23635107", 2024-09-17T12:21:59.429 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "r", 2024-09-17T12:21:59.429 INFO:teuthology.orchestra.run.smithi016.stdout: "current_period": "1d08ea89-3cff-4e41-85e4-5f3f9e1f57e0", 2024-09-17T12:21:59.429 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 1 2024-09-17T12:21:59.429 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:21:59.510 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:59.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: osdmap e62: 8 total, 8 up, 8 in 2024-09-17T12:21:59.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3486564170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T12:21:59.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T12:21:59.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T12:21:59.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T12:21:59.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T12:21:59.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:21:59.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:21:59.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T12:21:59.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T12:21:59.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T12:21:59.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T12:21:59.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T12:21:59.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T12:21:59.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:21:59.512 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:59.512 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: osdmap e62: 8 total, 8 up, 8 in 2024-09-17T12:21:59.512 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3486564170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T12:21:59.513 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.513 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.513 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T12:21:59.513 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T12:21:59.513 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T12:21:59.513 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T12:21:59.513 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.513 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:21:59.513 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:21:59.513 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T12:21:59.513 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T12:21:59.513 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.513 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T12:21:59.514 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T12:21:59.514 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T12:21:59.514 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T12:21:59.514 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.514 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:21:59.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:21:59.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: osdmap e62: 8 total, 8 up, 8 in 2024-09-17T12:21:59.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3486564170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T12:21:59.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T12:21:59.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T12:21:59.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T12:21:59.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T12:21:59.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:21:59.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:21:59.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T12:21:59.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T12:21:59.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T12:21:59.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T12:21:59.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T12:21:59.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T12:21:59.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:21:59.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:21:59 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:21:59.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T12:21:59.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:59] ENGINE Bus STOPPED 2024-09-17T12:21:59.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:59] ENGINE Bus STARTING 2024-09-17T12:21:59.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:59] ENGINE Serving on http://:::9283 2024-09-17T12:21:59.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:59] ENGINE Bus STARTED 2024-09-17T12:21:59.876 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:21:59 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:21:59] ENGINE Bus STOPPING 2024-09-17T12:22:00.138 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-09-17T12:22:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:22:00.410 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:00 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:00] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T12:22:00.410 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:00 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:00] ENGINE Bus STOPPED 2024-09-17T12:22:00.410 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:00 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:00] ENGINE Bus STARTING 2024-09-17T12:22:00.410 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:00 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:00] ENGINE Serving on http://:::9283 2024-09-17T12:22:00.410 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:00 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:00] ENGINE Bus STARTED 2024-09-17T12:22:00.683 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:00 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3486564170' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T12:22:00.683 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:00 smithi196 ceph-mon[27428]: osdmap e63: 8 total, 8 up, 8 in 2024-09-17T12:22:00.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:00 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3486564170' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T12:22:00.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:00 smithi016 ceph-mon[23719]: osdmap e63: 8 total, 8 up, 8 in 2024-09-17T12:22:00.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:00 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3486564170' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T12:22:00.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:00 smithi016 ceph-mon[34212]: osdmap e63: 8 total, 8 up, 8 in 2024-09-17T12:22:01.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:01 smithi016 ceph-mon[23719]: pgmap v39: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:01.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:01 smithi016 ceph-mon[23719]: osdmap e64: 8 total, 8 up, 8 in 2024-09-17T12:22:01.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:01 smithi016 ceph-mon[34212]: pgmap v39: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:01.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:01 smithi016 ceph-mon[34212]: osdmap e64: 8 total, 8 up, 8 in 2024-09-17T12:22:01.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:01 smithi196 ceph-mon[27428]: pgmap v39: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:01.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:01 smithi196 ceph-mon[27428]: osdmap e64: 8 total, 8 up, 8 in 2024-09-17T12:22:02.331 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:22:02.331 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "40e078a6-e638-40a2-ad95-465eda1bbe62", 2024-09-17T12:22:02.331 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default", 2024-09-17T12:22:02.331 INFO:teuthology.orchestra.run.smithi016.stdout: "api_name": "default", 2024-09-17T12:22:02.331 INFO:teuthology.orchestra.run.smithi016.stdout: "is_master": true, 2024-09-17T12:22:02.331 INFO:teuthology.orchestra.run.smithi016.stdout: "endpoints": [], 2024-09-17T12:22:02.331 INFO:teuthology.orchestra.run.smithi016.stdout: "hostnames": [], 2024-09-17T12:22:02.332 INFO:teuthology.orchestra.run.smithi016.stdout: "hostnames_s3website": [], 2024-09-17T12:22:02.332 INFO:teuthology.orchestra.run.smithi016.stdout: "master_zone": "", 2024-09-17T12:22:02.332 INFO:teuthology.orchestra.run.smithi016.stdout: "zones": [], 2024-09-17T12:22:02.332 INFO:teuthology.orchestra.run.smithi016.stdout: "placement_targets": [ 2024-09-17T12:22:02.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-17T12:22:02.332 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default-placement", 2024-09-17T12:22:02.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-17T12:22:02.332 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_classes": [] 2024-09-17T12:22:02.332 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:22:02.332 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:22:02.332 INFO:teuthology.orchestra.run.smithi016.stdout: "default_placement": "default-placement", 2024-09-17T12:22:02.332 INFO:teuthology.orchestra.run.smithi016.stdout: "realm_id": "d0fc053a-3d7f-40aa-976d-e3de23635107", 2024-09-17T12:22:02.332 INFO:teuthology.orchestra.run.smithi016.stdout: "sync_policy": { 2024-09-17T12:22:02.333 INFO:teuthology.orchestra.run.smithi016.stdout: "groups": [] 2024-09-17T12:22:02.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:22:02.333 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled_features": [ 2024-09-17T12:22:02.333 INFO:teuthology.orchestra.run.smithi016.stdout: "resharding" 2024-09-17T12:22:02.333 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-17T12:22:02.333 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:22:02.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:02 smithi016 ceph-mon[34212]: pgmap v41: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:02.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:02 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:02.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:02 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:02.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:02 smithi016 ceph-mon[23719]: pgmap v41: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:02.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:02 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:02.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:02 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:02.992 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-09-17T12:22:03.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:02 smithi196 ceph-mon[27428]: pgmap v41: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:03.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:02 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:03.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:02 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:22:03.691 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:22:03 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:22:03.279Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002405326s 2024-09-17T12:22:04.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:04 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:04.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:04 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:04.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:04 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:04.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:04 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:04.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:04 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:04.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:04 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:04.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:04 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:04.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:04 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:04.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:04 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:04.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:04 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:05.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:04 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:05.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:04 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:05.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:04 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:05.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:04 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:05.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:04 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:05.201 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:22:05.202 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "65fe8b73-96a1-435e-a27f-0db295d90ae8", 2024-09-17T12:22:05.202 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "z", 2024-09-17T12:22:05.202 INFO:teuthology.orchestra.run.smithi016.stdout: "domain_root": "z.rgw.meta:root", 2024-09-17T12:22:05.202 INFO:teuthology.orchestra.run.smithi016.stdout: "control_pool": "z.rgw.control", 2024-09-17T12:22:05.202 INFO:teuthology.orchestra.run.smithi016.stdout: "gc_pool": "z.rgw.log:gc", 2024-09-17T12:22:05.202 INFO:teuthology.orchestra.run.smithi016.stdout: "lc_pool": "z.rgw.log:lc", 2024-09-17T12:22:05.202 INFO:teuthology.orchestra.run.smithi016.stdout: "log_pool": "z.rgw.log", 2024-09-17T12:22:05.202 INFO:teuthology.orchestra.run.smithi016.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-09-17T12:22:05.202 INFO:teuthology.orchestra.run.smithi016.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-09-17T12:22:05.202 INFO:teuthology.orchestra.run.smithi016.stdout: "roles_pool": "z.rgw.meta:roles", 2024-09-17T12:22:05.202 INFO:teuthology.orchestra.run.smithi016.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-09-17T12:22:05.202 INFO:teuthology.orchestra.run.smithi016.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-09-17T12:22:05.202 INFO:teuthology.orchestra.run.smithi016.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: "otp_pool": "z.rgw.otp", 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: "system_key": { 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: "access_key": "", 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: "secret_key": "" 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: "placement_pools": [ 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "default-placement", 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: "val": { 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: "index_pool": "z.rgw.buckets.index", 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_classes": { 2024-09-17T12:22:05.203 INFO:teuthology.orchestra.run.smithi016.stdout: "STANDARD": { 2024-09-17T12:22:05.204 INFO:teuthology.orchestra.run.smithi016.stdout: "data_pool": "z.rgw.buckets.data" 2024-09-17T12:22:05.204 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:22:05.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:22:05.204 INFO:teuthology.orchestra.run.smithi016.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-09-17T12:22:05.204 INFO:teuthology.orchestra.run.smithi016.stdout: "index_type": 0, 2024-09-17T12:22:05.204 INFO:teuthology.orchestra.run.smithi016.stdout: "inline_data": true 2024-09-17T12:22:05.204 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:22:05.204 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:22:05.204 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:22:05.204 INFO:teuthology.orchestra.run.smithi016.stdout: "realm_id": "d0fc053a-3d7f-40aa-976d-e3de23635107", 2024-09-17T12:22:05.204 INFO:teuthology.orchestra.run.smithi016.stdout: "notif_pool": "z.rgw.log:notif" 2024-09-17T12:22:05.205 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:22:05.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:05 smithi016 ceph-mon[23719]: pgmap v42: 33 pgs: 33 active+clean; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 4 op/s 2024-09-17T12:22:05.859 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:05 smithi016 ceph-mon[34212]: pgmap v42: 33 pgs: 33 active+clean; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 4 op/s 2024-09-17T12:22:05.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-09-17T12:22:06.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:05 smithi196 ceph-mon[27428]: pgmap v42: 33 pgs: 33 active+clean; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 4 op/s 2024-09-17T12:22:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:22:06.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:22:06] "GET /metrics HTTP/1.1" 200 26201 "" "Prometheus/2.43.0" 2024-09-17T12:22:08.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:07 smithi196 ceph-mon[27428]: pgmap v43: 33 pgs: 33 active+clean; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-17T12:22:08.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:07 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:22:08.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:07 smithi016 ceph-mon[23719]: pgmap v43: 33 pgs: 33 active+clean; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-17T12:22:08.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:07 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:22:08.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:07 smithi016 ceph-mon[34212]: pgmap v43: 33 pgs: 33 active+clean; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-17T12:22:08.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:07 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:22:09.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:09 smithi016 ceph-mon[23719]: pgmap v44: 33 pgs: 33 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-09-17T12:22:09.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:09 smithi016 ceph-mon[23719]: osdmap e65: 8 total, 8 up, 8 in 2024-09-17T12:22:09.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:09 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3299938227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T12:22:09.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:09 smithi016 ceph-mon[34212]: pgmap v44: 33 pgs: 33 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-09-17T12:22:09.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:09 smithi016 ceph-mon[34212]: osdmap e65: 8 total, 8 up, 8 in 2024-09-17T12:22:09.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:09 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3299938227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T12:22:09.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:09 smithi196 ceph-mon[27428]: pgmap v44: 33 pgs: 33 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-09-17T12:22:09.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:09 smithi196 ceph-mon[27428]: osdmap e65: 8 total, 8 up, 8 in 2024-09-17T12:22:09.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:09 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3299938227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T12:22:10.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:10 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3299938227' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-17T12:22:10.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:10 smithi016 ceph-mon[23719]: osdmap e66: 8 total, 8 up, 8 in 2024-09-17T12:22:10.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:10 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3299938227' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-17T12:22:10.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:10 smithi016 ceph-mon[34212]: osdmap e66: 8 total, 8 up, 8 in 2024-09-17T12:22:10.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:10 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3299938227' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-17T12:22:10.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:10 smithi196 ceph-mon[27428]: osdmap e66: 8 total, 8 up, 8 in 2024-09-17T12:22:11.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:11 smithi016 ceph-mon[23719]: pgmap v47: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-09-17T12:22:11.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:11 smithi016 ceph-mon[23719]: osdmap e67: 8 total, 8 up, 8 in 2024-09-17T12:22:11.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:11 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3299938227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T12:22:11.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:11 smithi016 ceph-mon[34212]: pgmap v47: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-09-17T12:22:11.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:11 smithi016 ceph-mon[34212]: osdmap e67: 8 total, 8 up, 8 in 2024-09-17T12:22:11.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:11 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3299938227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T12:22:11.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:11 smithi196 ceph-mon[27428]: pgmap v47: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-09-17T12:22:11.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:11 smithi196 ceph-mon[27428]: osdmap e67: 8 total, 8 up, 8 in 2024-09-17T12:22:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:11 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3299938227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T12:22:12.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:12 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3299938227' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-17T12:22:12.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:12 smithi016 ceph-mon[23719]: osdmap e68: 8 total, 8 up, 8 in 2024-09-17T12:22:12.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:12 smithi016 ceph-mon[23719]: osdmap e69: 8 total, 8 up, 8 in 2024-09-17T12:22:12.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:12 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3401972308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T12:22:12.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:12 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3299938227' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-17T12:22:12.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:12 smithi016 ceph-mon[34212]: osdmap e68: 8 total, 8 up, 8 in 2024-09-17T12:22:12.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:12 smithi016 ceph-mon[34212]: osdmap e69: 8 total, 8 up, 8 in 2024-09-17T12:22:12.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:12 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3401972308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T12:22:12.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:12 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3299938227' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-17T12:22:12.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:12 smithi196 ceph-mon[27428]: osdmap e68: 8 total, 8 up, 8 in 2024-09-17T12:22:12.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:12 smithi196 ceph-mon[27428]: osdmap e69: 8 total, 8 up, 8 in 2024-09-17T12:22:12.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:12 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3401972308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T12:22:13.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:13 smithi196 ceph-mon[27428]: pgmap v50: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:13.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:13 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3401972308' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-17T12:22:13.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:13 smithi196 ceph-mon[27428]: osdmap e70: 8 total, 8 up, 8 in 2024-09-17T12:22:13.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:13 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3401972308' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T12:22:13.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:13 smithi016 ceph-mon[34212]: pgmap v50: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:13.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:13 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3401972308' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-17T12:22:13.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:13 smithi016 ceph-mon[34212]: osdmap e70: 8 total, 8 up, 8 in 2024-09-17T12:22:13.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:13 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3401972308' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T12:22:13.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:13 smithi016 ceph-mon[23719]: pgmap v50: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:13.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:13 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3401972308' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-17T12:22:13.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:13 smithi016 ceph-mon[23719]: osdmap e70: 8 total, 8 up, 8 in 2024-09-17T12:22:13.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:13 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3401972308' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T12:22:14.107 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:22:14.107 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "4e1a334a-43b2-4191-9462-0e127896c875", 2024-09-17T12:22:14.107 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 1, 2024-09-17T12:22:14.107 INFO:teuthology.orchestra.run.smithi016.stdout: "predecessor_uuid": "1d08ea89-3cff-4e41-85e4-5f3f9e1f57e0", 2024-09-17T12:22:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "sync_status": [], 2024-09-17T12:22:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "period_map": { 2024-09-17T12:22:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "4e1a334a-43b2-4191-9462-0e127896c875", 2024-09-17T12:22:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "zonegroups": [ 2024-09-17T12:22:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-17T12:22:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "40e078a6-e638-40a2-ad95-465eda1bbe62", 2024-09-17T12:22:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default", 2024-09-17T12:22:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "api_name": "default", 2024-09-17T12:22:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "is_master": true, 2024-09-17T12:22:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "endpoints": [], 2024-09-17T12:22:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "hostnames": [], 2024-09-17T12:22:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "hostnames_s3website": [], 2024-09-17T12:22:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "master_zone": "65fe8b73-96a1-435e-a27f-0db295d90ae8", 2024-09-17T12:22:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "zones": [ 2024-09-17T12:22:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-17T12:22:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "65fe8b73-96a1-435e-a27f-0db295d90ae8", 2024-09-17T12:22:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "z", 2024-09-17T12:22:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "endpoints": [], 2024-09-17T12:22:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "log_meta": false, 2024-09-17T12:22:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "log_data": false, 2024-09-17T12:22:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "bucket_index_max_shards": 11, 2024-09-17T12:22:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "read_only": false, 2024-09-17T12:22:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_type": "", 2024-09-17T12:22:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "sync_from_all": true, 2024-09-17T12:22:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "sync_from": [], 2024-09-17T12:22:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_zone": "", 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: "supported_features": [ 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: "compress-encrypted", 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: "resharding" 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: "placement_targets": [ 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default-placement", 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_classes": [ 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: "STANDARD" 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-17T12:22:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "default_placement": "default-placement", 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "realm_id": "d0fc053a-3d7f-40aa-976d-e3de23635107", 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "sync_policy": { 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "groups": [] 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled_features": [ 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "resharding" 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "short_zone_ids": [ 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-17T12:22:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "65fe8b73-96a1-435e-a27f-0db295d90ae8", 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 459932339 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "master_zonegroup": "40e078a6-e638-40a2-ad95-465eda1bbe62", 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "master_zone": "65fe8b73-96a1-435e-a27f-0db295d90ae8", 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "period_config": { 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "bucket_quota": { 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": false, 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "check_on_raw": false, 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "max_size": -1, 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "max_size_kb": 0, 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "max_objects": -1 2024-09-17T12:22:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:22:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "user_quota": { 2024-09-17T12:22:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": false, 2024-09-17T12:22:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "check_on_raw": false, 2024-09-17T12:22:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max_size": -1, 2024-09-17T12:22:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max_size_kb": 0, 2024-09-17T12:22:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max_objects": -1 2024-09-17T12:22:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:22:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "user_ratelimit": { 2024-09-17T12:22:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max_read_ops": 0, 2024-09-17T12:22:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max_write_ops": 0, 2024-09-17T12:22:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max_read_bytes": 0, 2024-09-17T12:22:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max_write_bytes": 0, 2024-09-17T12:22:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": false 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "bucket_ratelimit": { 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max_read_ops": 0, 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max_write_ops": 0, 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max_read_bytes": 0, 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max_write_bytes": 0, 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": false 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "anonymous_ratelimit": { 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max_read_ops": 0, 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max_write_ops": 0, 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max_read_bytes": 0, 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max_write_bytes": 0, 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": false 2024-09-17T12:22:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:22:14.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:22:14.115 INFO:teuthology.orchestra.run.smithi016.stdout: "realm_id": "d0fc053a-3d7f-40aa-976d-e3de23635107", 2024-09-17T12:22:14.115 INFO:teuthology.orchestra.run.smithi016.stdout: "realm_name": "r", 2024-09-17T12:22:14.115 INFO:teuthology.orchestra.run.smithi016.stdout: "realm_epoch": 2 2024-09-17T12:22:14.115 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:22:14.804 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-09-17T12:22:15.014 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:22:15.347 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:15 smithi016 ceph-mon[34212]: pgmap v53: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-09-17T12:22:15.348 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:15 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3401972308' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T12:22:15.348 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:15 smithi016 ceph-mon[34212]: osdmap e71: 8 total, 8 up, 8 in 2024-09-17T12:22:15.348 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:15 smithi016 ceph-mon[23719]: pgmap v53: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-09-17T12:22:15.348 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:15 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3401972308' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T12:22:15.348 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:15 smithi016 ceph-mon[23719]: osdmap e71: 8 total, 8 up, 8 in 2024-09-17T12:22:15.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:15 smithi196 ceph-mon[27428]: pgmap v53: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-09-17T12:22:15.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:15 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3401972308' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T12:22:15.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:15 smithi196 ceph-mon[27428]: osdmap e71: 8 total, 8 up, 8 in 2024-09-17T12:22:16.321 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:22:16] "GET /metrics HTTP/1.1" 200 28523 "" "Prometheus/2.43.0" 2024-09-17T12:22:16.710 INFO:teuthology.orchestra.run.smithi016.stdout:Scheduled rgw.foo update... 2024-09-17T12:22:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:17 smithi196 ceph-mon[27428]: pgmap v55: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 219 B/s rd, 439 B/s wr, 1 op/s 2024-09-17T12:22:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:17 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:17 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:22:17.342 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:17 smithi016 ceph-mon[23719]: pgmap v55: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 219 B/s rd, 439 B/s wr, 1 op/s 2024-09-17T12:22:17.342 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:17 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:17.343 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:17 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:22:17.343 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:17 smithi016 ceph-mon[34212]: pgmap v55: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 219 B/s rd, 439 B/s wr, 1 op/s 2024-09-17T12:22:17.343 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:17 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:17.343 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:17 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:22:17.365 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph orch apply rgw smpl' 2024-09-17T12:22:18.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:18 smithi016 ceph-mon[34212]: from='client.24593 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:22:18.208 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:18 smithi016 ceph-mon[34212]: Saving service rgw.foo spec with placement count:2 2024-09-17T12:22:18.208 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:18 smithi016 ceph-mon[23719]: from='client.24593 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:22:18.208 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:18 smithi016 ceph-mon[23719]: Saving service rgw.foo spec with placement count:2 2024-09-17T12:22:18.221 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:22:18.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:18 smithi196 ceph-mon[27428]: from='client.24593 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:22:18.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:18 smithi196 ceph-mon[27428]: Saving service rgw.foo spec with placement count:2 2024-09-17T12:22:19.304 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:19 smithi016 ceph-mon[23719]: pgmap v56: 129 pgs: 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-09-17T12:22:19.304 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:19 smithi016 ceph-mon[34212]: pgmap v56: 129 pgs: 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-09-17T12:22:19.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:19 smithi196 ceph-mon[27428]: pgmap v56: 129 pgs: 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-09-17T12:22:19.997 INFO:teuthology.orchestra.run.smithi016.stdout:Scheduled rgw.smpl update... 2024-09-17T12:22:21.018 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph osd pool create foo' 2024-09-17T12:22:21.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:20 smithi016 ceph-mon[23719]: pgmap v57: 129 pgs: 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-09-17T12:22:21.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:20 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:21.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:20 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:21.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:20 smithi016 ceph-mon[23719]: from='client.24599 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:22:21.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:20 smithi016 ceph-mon[23719]: Saving service rgw.smpl spec with placement count:2 2024-09-17T12:22:21.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:20 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:21.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:20 smithi016 ceph-mon[34212]: pgmap v57: 129 pgs: 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-09-17T12:22:21.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:20 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:21.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:20 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:21.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:20 smithi016 ceph-mon[34212]: from='client.24599 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:22:21.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:20 smithi016 ceph-mon[34212]: Saving service rgw.smpl spec with placement count:2 2024-09-17T12:22:21.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:20 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:21.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:20 smithi196 ceph-mon[27428]: pgmap v57: 129 pgs: 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-09-17T12:22:21.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:20 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:21.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:20 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:21.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:20 smithi196 ceph-mon[27428]: from='client.24599 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:22:21.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:20 smithi196 ceph-mon[27428]: Saving service rgw.smpl spec with placement count:2 2024-09-17T12:22:21.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:20 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:22:22.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:22:22.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:22.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:22.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:22.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:22.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:22.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:22.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:22.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T12:22:22.140 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:22.140 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:22.140 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:22:22.140 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:22.140 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:22.140 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:22.140 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:22.140 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:22.140 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:22.140 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:22.141 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T12:22:22.141 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:22.141 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:21 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:22.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:21 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:22:22.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:21 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:22.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:21 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:22.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:21 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:22.252 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:21 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:22.252 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:21 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:22.252 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:21 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:22.252 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:21 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:22.252 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:21 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T12:22:22.253 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:21 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:22.253 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:21 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:23.180 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:22 smithi016 ceph-mon[23719]: pgmap v58: 129 pgs: 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-17T12:22:23.180 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:22 smithi016 ceph-mon[23719]: Deploying daemon rgw.foo.smithi196.wvbocw on smithi196 2024-09-17T12:22:23.181 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:22 smithi016 ceph-mon[34212]: pgmap v58: 129 pgs: 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-17T12:22:23.181 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:22 smithi016 ceph-mon[34212]: Deploying daemon rgw.foo.smithi196.wvbocw on smithi196 2024-09-17T12:22:23.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:22 smithi196 ceph-mon[27428]: pgmap v58: 129 pgs: 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-17T12:22:23.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:22 smithi196 ceph-mon[27428]: Deploying daemon rgw.foo.smithi196.wvbocw on smithi196 2024-09-17T12:22:24.153 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'foo' created 2024-09-17T12:22:24.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:24 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1625883678' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T12:22:24.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:24 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1625883678' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T12:22:24.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:24 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1625883678' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T12:22:24.894 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'rbd pool init foo' 2024-09-17T12:22:25.425 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[23719]: pgmap v59: 129 pgs: 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-17T12:22:25.425 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1625883678' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-17T12:22:25.425 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[23719]: osdmap e72: 8 total, 8 up, 8 in 2024-09-17T12:22:25.425 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:25.425 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:25.425 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:25.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:25.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:25.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T12:22:25.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:25.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:25.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[23719]: Deploying daemon rgw.foo.smithi016.tquwud on smithi016 2024-09-17T12:22:25.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[23719]: osdmap e73: 8 total, 8 up, 8 in 2024-09-17T12:22:25.426 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[34212]: pgmap v59: 129 pgs: 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-17T12:22:25.426 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1625883678' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-17T12:22:25.426 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[34212]: osdmap e72: 8 total, 8 up, 8 in 2024-09-17T12:22:25.427 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:25.427 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:25.427 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:25.427 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:25.427 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:25.427 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T12:22:25.427 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:25.427 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:25.427 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[34212]: Deploying daemon rgw.foo.smithi016.tquwud on smithi016 2024-09-17T12:22:25.427 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:25 smithi016 ceph-mon[34212]: osdmap e73: 8 total, 8 up, 8 in 2024-09-17T12:22:25.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:25 smithi196 ceph-mon[27428]: pgmap v59: 129 pgs: 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-17T12:22:25.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:25 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1625883678' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-17T12:22:25.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:25 smithi196 ceph-mon[27428]: osdmap e72: 8 total, 8 up, 8 in 2024-09-17T12:22:25.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:25 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:25.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:25 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:25.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:25 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:25.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:25 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:25.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:25 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:25.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:25 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T12:22:25.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:25 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:25.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:25 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:25.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:25 smithi196 ceph-mon[27428]: Deploying daemon rgw.foo.smithi016.tquwud on smithi016 2024-09-17T12:22:25.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:25 smithi196 ceph-mon[27428]: osdmap e73: 8 total, 8 up, 8 in 2024-09-17T12:22:26.035 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:22:26.298 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:22:26] "GET /metrics HTTP/1.1" 200 35424 "" "Prometheus/2.43.0" 2024-09-17T12:22:27.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:27 smithi016 ceph-mon[23719]: pgmap v62: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:27.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:27 smithi016 ceph-mon[23719]: osdmap e74: 8 total, 8 up, 8 in 2024-09-17T12:22:27.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:27 smithi016 ceph-mon[23719]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T12:22:27.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:27 smithi016 ceph-mon[34212]: pgmap v62: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:27.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:27 smithi016 ceph-mon[34212]: osdmap e74: 8 total, 8 up, 8 in 2024-09-17T12:22:27.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:27 smithi016 ceph-mon[34212]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T12:22:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:27 smithi196 ceph-mon[27428]: pgmap v62: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:27 smithi196 ceph-mon[27428]: osdmap e74: 8 total, 8 up, 8 in 2024-09-17T12:22:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:27 smithi196 ceph-mon[27428]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T12:22:28.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:28 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/4212667606' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T12:22:28.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:28 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/4212667606' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T12:22:28.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:28 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/4212667606' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T12:22:29.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: pgmap v64: 161 pgs: 161 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.5 KiB/s wr, 186 op/s 2024-09-17T12:22:29.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/4212667606' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-17T12:22:29.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: osdmap e75: 8 total, 8 up, 8 in 2024-09-17T12:22:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: Saving service rgw.foo spec with placement count:2 2024-09-17T12:22:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T12:22:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:29.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[23719]: Deploying daemon rgw.smpl.smithi016.qdkyfn on smithi016 2024-09-17T12:22:29.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: pgmap v64: 161 pgs: 161 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.5 KiB/s wr, 186 op/s 2024-09-17T12:22:29.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/4212667606' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-17T12:22:29.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: osdmap e75: 8 total, 8 up, 8 in 2024-09-17T12:22:29.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: Saving service rgw.foo spec with placement count:2 2024-09-17T12:22:29.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:29.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:29.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T12:22:29.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:29.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:29 smithi016 ceph-mon[34212]: Deploying daemon rgw.smpl.smithi016.qdkyfn on smithi016 2024-09-17T12:22:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: pgmap v64: 161 pgs: 161 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.5 KiB/s wr, 186 op/s 2024-09-17T12:22:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/4212667606' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-17T12:22:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: osdmap e75: 8 total, 8 up, 8 in 2024-09-17T12:22:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: Saving service rgw.foo spec with placement count:2 2024-09-17T12:22:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:29.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:29.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T12:22:29.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:29.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:29.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:29 smithi196 ceph-mon[27428]: Deploying daemon rgw.smpl.smithi016.qdkyfn on smithi016 2024-09-17T12:22:30.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:30 smithi016 ceph-mon[23719]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T12:22:30.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:30 smithi016 ceph-mon[34212]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T12:22:30.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:30 smithi196 ceph-mon[27428]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T12:22:30.799 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph orch apply iscsi foo u p' 2024-09-17T12:22:31.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:31 smithi016 ceph-mon[23719]: pgmap v67: 161 pgs: 161 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 7.2 KiB/s wr, 245 op/s 2024-09-17T12:22:31.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:31 smithi016 ceph-mon[23719]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T12:22:31.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:31 smithi016 ceph-mon[23719]: Cluster is now healthy 2024-09-17T12:22:31.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:31 smithi016 ceph-mon[23719]: osdmap e77: 8 total, 8 up, 8 in 2024-09-17T12:22:31.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:31 smithi016 ceph-mon[34212]: pgmap v67: 161 pgs: 161 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 7.2 KiB/s wr, 245 op/s 2024-09-17T12:22:31.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:31 smithi016 ceph-mon[34212]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T12:22:31.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:31 smithi016 ceph-mon[34212]: Cluster is now healthy 2024-09-17T12:22:31.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:31 smithi016 ceph-mon[34212]: osdmap e77: 8 total, 8 up, 8 in 2024-09-17T12:22:31.464 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:22:31.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:31 smithi196 ceph-mon[27428]: pgmap v67: 161 pgs: 161 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 7.2 KiB/s wr, 245 op/s 2024-09-17T12:22:31.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:31 smithi196 ceph-mon[27428]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T12:22:31.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:31 smithi196 ceph-mon[27428]: Cluster is now healthy 2024-09-17T12:22:31.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:31 smithi196 ceph-mon[27428]: osdmap e77: 8 total, 8 up, 8 in 2024-09-17T12:22:32.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:32.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:32.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T12:22:32.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:32.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[23719]: Deploying daemon rgw.smpl.smithi196.ghtvuz on smithi196 2024-09-17T12:22:32.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:32.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:32.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T12:22:32.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:32.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[34212]: Deploying daemon rgw.smpl.smithi196.ghtvuz on smithi196 2024-09-17T12:22:32.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:32 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:32 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:32 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:32 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:32 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:32.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:32 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T12:22:32.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:32 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T12:22:32.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:32 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:32.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:32 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:32.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:32 smithi196 ceph-mon[27428]: Deploying daemon rgw.smpl.smithi196.ghtvuz on smithi196 2024-09-17T12:22:32.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:32 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:33.273 INFO:teuthology.orchestra.run.smithi016.stdout:Scheduled iscsi.foo update... 2024-09-17T12:22:33.670 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:33 smithi016 ceph-mon[34212]: pgmap v69: 161 pgs: 161 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 5.9 KiB/s wr, 201 op/s 2024-09-17T12:22:33.671 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:33 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:33.671 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:33 smithi016 ceph-mon[23719]: pgmap v69: 161 pgs: 161 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 5.9 KiB/s wr, 201 op/s 2024-09-17T12:22:33.671 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:33 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:33.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:33 smithi196 ceph-mon[27428]: pgmap v69: 161 pgs: 161 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 5.9 KiB/s wr, 201 op/s 2024-09-17T12:22:33.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:33 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.057 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'sleep 120' 2024-09-17T12:22:34.297 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:22:34.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[23719]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:22:34.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[23719]: Saving service iscsi.foo spec with placement count:1 2024-09-17T12:22:34.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:22:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[34212]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:22:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[34212]: Saving service iscsi.foo spec with placement count:1 2024-09-17T12:22:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:34 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:22:34.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:34 smithi196 ceph-mon[27428]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:22:34.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:34 smithi196 ceph-mon[27428]: Saving service iscsi.foo spec with placement count:1 2024-09-17T12:22:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:34 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:34 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:34 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:34 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:34 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:34.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:34 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:22:35.593 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:35 smithi016 ceph-mon[34212]: pgmap v70: 161 pgs: 161 active+clean; 584 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 215 KiB/s rd, 682 B/s wr, 331 op/s 2024-09-17T12:22:35.593 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:35 smithi016 ceph-mon[34212]: Saving service rgw.smpl spec with placement count:2 2024-09-17T12:22:35.593 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:35 smithi016 ceph-mon[23719]: pgmap v70: 161 pgs: 161 active+clean; 584 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 215 KiB/s rd, 682 B/s wr, 331 op/s 2024-09-17T12:22:35.593 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:35 smithi016 ceph-mon[23719]: Saving service rgw.smpl spec with placement count:2 2024-09-17T12:22:35.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:35 smithi196 ceph-mon[27428]: pgmap v70: 161 pgs: 161 active+clean; 584 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 215 KiB/s rd, 682 B/s wr, 331 op/s 2024-09-17T12:22:35.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:35 smithi196 ceph-mon[27428]: Saving service rgw.smpl spec with placement count:2 2024-09-17T12:22:36.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:22:36] "GET /metrics HTTP/1.1" 200 35424 "" "Prometheus/2.43.0" 2024-09-17T12:22:37.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:37 smithi016 ceph-mon[23719]: pgmap v71: 161 pgs: 161 active+clean; 584 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 542 B/s wr, 263 op/s 2024-09-17T12:22:37.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:37 smithi016 ceph-mon[23719]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T12:22:37.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:37 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:22:37.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:37 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:37 smithi016 ceph-mon[34212]: pgmap v71: 161 pgs: 161 active+clean; 584 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 542 B/s wr, 263 op/s 2024-09-17T12:22:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:37 smithi016 ceph-mon[34212]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T12:22:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:37 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:22:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:37 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:37.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:37 smithi196 ceph-mon[27428]: pgmap v71: 161 pgs: 161 active+clean; 584 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 542 B/s wr, 263 op/s 2024-09-17T12:22:37.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:37 smithi196 ceph-mon[27428]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T12:22:37.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:37 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:22:37.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:37 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:38] ENGINE Bus STOPPING 2024-09-17T12:22:38.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:22:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:22:38.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T12:22:38.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:38.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:38.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:38.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:22:38.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:22:38.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T12:22:38.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:38.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:38 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:38 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:38 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:38 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:38 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:38 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:38 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:38 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:38 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:38 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:22:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:38 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:22:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:38 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T12:22:38.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:38 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:38.985 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T12:22:38.985 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:38] ENGINE Bus STOPPED 2024-09-17T12:22:38.985 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:38] ENGINE Bus STARTING 2024-09-17T12:22:38.985 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:38] ENGINE Serving on http://:::9283 2024-09-17T12:22:38.985 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:38] ENGINE Bus STARTED 2024-09-17T12:22:38.985 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:38] ENGINE Bus STOPPING 2024-09-17T12:22:39.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:39 smithi016 ceph-mon[23719]: pgmap v73: 161 pgs: 161 active+clean; 584 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 241 KiB/s rd, 639 B/s wr, 373 op/s 2024-09-17T12:22:39.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:39 smithi016 ceph-mon[23719]: Checking dashboard <-> RGW credentials 2024-09-17T12:22:39.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:39 smithi016 ceph-mon[23719]: Deploying daemon iscsi.foo.smithi016.qaboca on smithi016 2024-09-17T12:22:39.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:39 smithi016 ceph-mon[34212]: pgmap v73: 161 pgs: 161 active+clean; 584 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 241 KiB/s rd, 639 B/s wr, 373 op/s 2024-09-17T12:22:39.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:39 smithi016 ceph-mon[34212]: Checking dashboard <-> RGW credentials 2024-09-17T12:22:39.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:39 smithi016 ceph-mon[34212]: Deploying daemon iscsi.foo.smithi016.qaboca on smithi016 2024-09-17T12:22:39.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T12:22:39.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:39] ENGINE Bus STOPPED 2024-09-17T12:22:39.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:39] ENGINE Bus STARTING 2024-09-17T12:22:39.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:39] ENGINE Serving on http://:::9283 2024-09-17T12:22:39.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:39] ENGINE Bus STARTED 2024-09-17T12:22:39.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:39 smithi196 ceph-mon[27428]: pgmap v73: 161 pgs: 161 active+clean; 584 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 241 KiB/s rd, 639 B/s wr, 373 op/s 2024-09-17T12:22:39.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:39 smithi196 ceph-mon[27428]: Checking dashboard <-> RGW credentials 2024-09-17T12:22:39.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:39 smithi196 ceph-mon[27428]: Deploying daemon iscsi.foo.smithi016.qaboca on smithi016 2024-09-17T12:22:41.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:41 smithi016 ceph-mon[23719]: pgmap v74: 161 pgs: 161 active+clean; 584 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 203 KiB/s rd, 536 B/s wr, 313 op/s 2024-09-17T12:22:41.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:41 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:41.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:41 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:41.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:41 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:41.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:41 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:41.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:41 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:22:41.749 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:41 smithi016 ceph-mon[34212]: pgmap v74: 161 pgs: 161 active+clean; 584 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 203 KiB/s rd, 536 B/s wr, 313 op/s 2024-09-17T12:22:41.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:41 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:41.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:41 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:41.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:41 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:41.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:41 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:41.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:41 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:22:41.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:41 smithi196 ceph-mon[27428]: pgmap v74: 161 pgs: 161 active+clean; 584 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 203 KiB/s rd, 536 B/s wr, 313 op/s 2024-09-17T12:22:41.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:41 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:41.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:41 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:41.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:41 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:41.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:41 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:41.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:41 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:22:42.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:42 smithi196 ceph-mon[27428]: Checking pool "foo" exists for service iscsi.foo 2024-09-17T12:22:42.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:42 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1280516528' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T12:22:42.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:42 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:42.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:42 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/279765743' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3211434426"}]: dispatch 2024-09-17T12:22:42.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:42 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3211434426"}]: dispatch 2024-09-17T12:22:42.783 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:42 smithi016 ceph-mon[34212]: Checking pool "foo" exists for service iscsi.foo 2024-09-17T12:22:42.783 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:42 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1280516528' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T12:22:42.783 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:42 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:42.783 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:42 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/279765743' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3211434426"}]: dispatch 2024-09-17T12:22:42.783 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:42 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3211434426"}]: dispatch 2024-09-17T12:22:42.783 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:42 smithi016 ceph-mon[23719]: Checking pool "foo" exists for service iscsi.foo 2024-09-17T12:22:42.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:42 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1280516528' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T12:22:42.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:42 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:42.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:42 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/279765743' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3211434426"}]: dispatch 2024-09-17T12:22:42.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:42 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3211434426"}]: dispatch 2024-09-17T12:22:43.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:43 smithi196 ceph-mon[27428]: pgmap v75: 161 pgs: 161 active+clean; 584 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 511 B/s wr, 298 op/s 2024-09-17T12:22:43.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:43 smithi196 ceph-mon[27428]: mgrmap e22: y(active, since 110s), standbys: x 2024-09-17T12:22:43.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:43 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3211434426"}]': finished 2024-09-17T12:22:43.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:43 smithi196 ceph-mon[27428]: osdmap e79: 8 total, 8 up, 8 in 2024-09-17T12:22:43.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:43 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3802052574' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1225259398"}]: dispatch 2024-09-17T12:22:43.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:43 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1225259398"}]: dispatch 2024-09-17T12:22:43.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:43 smithi016 ceph-mon[23719]: pgmap v75: 161 pgs: 161 active+clean; 584 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 511 B/s wr, 298 op/s 2024-09-17T12:22:43.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:43 smithi016 ceph-mon[23719]: mgrmap e22: y(active, since 110s), standbys: x 2024-09-17T12:22:43.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:43 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3211434426"}]': finished 2024-09-17T12:22:43.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:43 smithi016 ceph-mon[23719]: osdmap e79: 8 total, 8 up, 8 in 2024-09-17T12:22:43.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:43 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3802052574' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1225259398"}]: dispatch 2024-09-17T12:22:43.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:43 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1225259398"}]: dispatch 2024-09-17T12:22:43.785 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:43 smithi016 ceph-mon[34212]: pgmap v75: 161 pgs: 161 active+clean; 584 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 511 B/s wr, 298 op/s 2024-09-17T12:22:43.786 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:43 smithi016 ceph-mon[34212]: mgrmap e22: y(active, since 110s), standbys: x 2024-09-17T12:22:43.786 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:43 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3211434426"}]': finished 2024-09-17T12:22:43.786 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:43 smithi016 ceph-mon[34212]: osdmap e79: 8 total, 8 up, 8 in 2024-09-17T12:22:43.786 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:43 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3802052574' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1225259398"}]: dispatch 2024-09-17T12:22:43.786 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:43 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1225259398"}]: dispatch 2024-09-17T12:22:45.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:44 smithi196 ceph-mon[27428]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 511 B/s wr, 129 op/s 2024-09-17T12:22:45.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:44 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1225259398"}]': finished 2024-09-17T12:22:45.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:44 smithi196 ceph-mon[27428]: osdmap e80: 8 total, 8 up, 8 in 2024-09-17T12:22:45.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:44 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1959041985' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1310624606"}]: dispatch 2024-09-17T12:22:45.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:44 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:45.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:44 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:45.045 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-mon[23719]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 511 B/s wr, 129 op/s 2024-09-17T12:22:45.045 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1225259398"}]': finished 2024-09-17T12:22:45.045 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-mon[23719]: osdmap e80: 8 total, 8 up, 8 in 2024-09-17T12:22:45.045 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1959041985' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1310624606"}]: dispatch 2024-09-17T12:22:45.045 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:45.045 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:45.045 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-mon[34212]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 511 B/s wr, 129 op/s 2024-09-17T12:22:45.046 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1225259398"}]': finished 2024-09-17T12:22:45.046 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-mon[34212]: osdmap e80: 8 total, 8 up, 8 in 2024-09-17T12:22:45.046 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1959041985' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1310624606"}]: dispatch 2024-09-17T12:22:45.046 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:45.046 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:45.046 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:44] ENGINE Bus STOPPING 2024-09-17T12:22:45.046 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T12:22:45.046 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:44] ENGINE Bus STOPPED 2024-09-17T12:22:45.046 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:44 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:44] ENGINE Bus STARTING 2024-09-17T12:22:45.046 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:45] ENGINE Serving on http://:::9283 2024-09-17T12:22:45.047 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:22:45] ENGINE Bus STARTED 2024-09-17T12:22:46.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1959041985' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1310624606"}]': finished 2024-09-17T12:22:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: osdmap e81: 8 total, 8 up, 8 in 2024-09-17T12:22:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T12:22:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T12:22:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: Adding iSCSI gateway http://:@172.21.15.16:5000 to Dashboard 2024-09-17T12:22:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T12:22:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T12:22:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi016"}]: dispatch 2024-09-17T12:22:46.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi016"}]: dispatch 2024-09-17T12:22:46.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:22:46.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1543573009' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/843792520"}]: dispatch 2024-09-17T12:22:46.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/843792520"}]: dispatch 2024-09-17T12:22:46.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1959041985' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1310624606"}]': finished 2024-09-17T12:22:46.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: osdmap e81: 8 total, 8 up, 8 in 2024-09-17T12:22:46.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: Adding iSCSI gateway http://:@172.21.15.16:5000 to Dashboard 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi016"}]: dispatch 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi016"}]: dispatch 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:22:46.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1543573009' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/843792520"}]: dispatch 2024-09-17T12:22:46.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:45 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/843792520"}]: dispatch 2024-09-17T12:22:46.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1959041985' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1310624606"}]': finished 2024-09-17T12:22:46.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: osdmap e81: 8 total, 8 up, 8 in 2024-09-17T12:22:46.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:46.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:46.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T12:22:46.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T12:22:46.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: Adding iSCSI gateway http://:@172.21.15.16:5000 to Dashboard 2024-09-17T12:22:46.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T12:22:46.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T12:22:46.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi016"}]: dispatch 2024-09-17T12:22:46.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi016"}]: dispatch 2024-09-17T12:22:46.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:46.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:22:46.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1543573009' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/843792520"}]: dispatch 2024-09-17T12:22:46.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:45 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/843792520"}]: dispatch 2024-09-17T12:22:46.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:22:46] "GET /metrics HTTP/1.1" 200 38236 "" "Prometheus/2.43.0" 2024-09-17T12:22:47.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:46 smithi016 ceph-mon[23719]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-17T12:22:47.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:46 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/843792520"}]': finished 2024-09-17T12:22:47.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:46 smithi016 ceph-mon[23719]: osdmap e82: 8 total, 8 up, 8 in 2024-09-17T12:22:47.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:46 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3825628351' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3411803517"}]: dispatch 2024-09-17T12:22:47.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:46 smithi016 ceph-mon[34212]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-17T12:22:47.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:46 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/843792520"}]': finished 2024-09-17T12:22:47.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:46 smithi016 ceph-mon[34212]: osdmap e82: 8 total, 8 up, 8 in 2024-09-17T12:22:47.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:46 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3825628351' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3411803517"}]: dispatch 2024-09-17T12:22:47.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:46 smithi196 ceph-mon[27428]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-17T12:22:47.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:46 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/843792520"}]': finished 2024-09-17T12:22:47.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:46 smithi196 ceph-mon[27428]: osdmap e82: 8 total, 8 up, 8 in 2024-09-17T12:22:47.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:46 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3825628351' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3411803517"}]: dispatch 2024-09-17T12:22:48.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:47 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3825628351' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3411803517"}]': finished 2024-09-17T12:22:48.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:47 smithi016 ceph-mon[23719]: osdmap e83: 8 total, 8 up, 8 in 2024-09-17T12:22:48.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:47 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/23890428' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1179446893"}]: dispatch 2024-09-17T12:22:48.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:47 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1179446893"}]: dispatch 2024-09-17T12:22:48.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:47 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3825628351' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3411803517"}]': finished 2024-09-17T12:22:48.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:47 smithi016 ceph-mon[34212]: osdmap e83: 8 total, 8 up, 8 in 2024-09-17T12:22:48.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:47 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/23890428' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1179446893"}]: dispatch 2024-09-17T12:22:48.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:47 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1179446893"}]: dispatch 2024-09-17T12:22:48.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:47 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3825628351' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3411803517"}]': finished 2024-09-17T12:22:48.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:47 smithi196 ceph-mon[27428]: osdmap e83: 8 total, 8 up, 8 in 2024-09-17T12:22:48.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:47 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/23890428' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1179446893"}]: dispatch 2024-09-17T12:22:48.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:47 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1179446893"}]: dispatch 2024-09-17T12:22:49.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:48 smithi196 ceph-mon[27428]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:22:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:48 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1179446893"}]': finished 2024-09-17T12:22:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:48 smithi196 ceph-mon[27428]: osdmap e84: 8 total, 8 up, 8 in 2024-09-17T12:22:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:48 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2254977073' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3212434883"}]: dispatch 2024-09-17T12:22:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:48 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3212434883"}]: dispatch 2024-09-17T12:22:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:48 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:48 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:48 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:48 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:48 smithi196 ceph-mon[27428]: Checking dashboard <-> RGW credentials 2024-09-17T12:22:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:48 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:49.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[23719]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:22:49.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1179446893"}]': finished 2024-09-17T12:22:49.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[23719]: osdmap e84: 8 total, 8 up, 8 in 2024-09-17T12:22:49.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2254977073' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3212434883"}]: dispatch 2024-09-17T12:22:49.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3212434883"}]: dispatch 2024-09-17T12:22:49.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:49.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:49.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:49.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:49.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[23719]: Checking dashboard <-> RGW credentials 2024-09-17T12:22:49.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:49.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[34212]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:22:49.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1179446893"}]': finished 2024-09-17T12:22:49.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[34212]: osdmap e84: 8 total, 8 up, 8 in 2024-09-17T12:22:49.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2254977073' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3212434883"}]: dispatch 2024-09-17T12:22:49.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3212434883"}]: dispatch 2024-09-17T12:22:49.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:49.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:49.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:22:49.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:22:49.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[34212]: Checking dashboard <-> RGW credentials 2024-09-17T12:22:49.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:48 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:22:50.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:49 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3212434883"}]': finished 2024-09-17T12:22:50.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:49 smithi196 ceph-mon[27428]: osdmap e85: 8 total, 8 up, 8 in 2024-09-17T12:22:50.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:49 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2780738555' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3212434883"}]: dispatch 2024-09-17T12:22:50.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:49 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3212434883"}]: dispatch 2024-09-17T12:22:50.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:49 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3212434883"}]': finished 2024-09-17T12:22:50.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:49 smithi016 ceph-mon[23719]: osdmap e85: 8 total, 8 up, 8 in 2024-09-17T12:22:50.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:49 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2780738555' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3212434883"}]: dispatch 2024-09-17T12:22:50.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:49 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3212434883"}]: dispatch 2024-09-17T12:22:50.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:49 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3212434883"}]': finished 2024-09-17T12:22:50.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:49 smithi016 ceph-mon[34212]: osdmap e85: 8 total, 8 up, 8 in 2024-09-17T12:22:50.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:49 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2780738555' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3212434883"}]: dispatch 2024-09-17T12:22:50.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:49 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3212434883"}]: dispatch 2024-09-17T12:22:51.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:50 smithi196 ceph-mon[27428]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:22:51.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:50 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3212434883"}]': finished 2024-09-17T12:22:51.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:50 smithi196 ceph-mon[27428]: osdmap e86: 8 total, 8 up, 8 in 2024-09-17T12:22:51.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:50 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3432043133' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/29394240"}]: dispatch 2024-09-17T12:22:51.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:50 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/29394240"}]: dispatch 2024-09-17T12:22:51.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:50 smithi016 ceph-mon[23719]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:22:51.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:50 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3212434883"}]': finished 2024-09-17T12:22:51.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:50 smithi016 ceph-mon[23719]: osdmap e86: 8 total, 8 up, 8 in 2024-09-17T12:22:51.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:50 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3432043133' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/29394240"}]: dispatch 2024-09-17T12:22:51.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:50 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/29394240"}]: dispatch 2024-09-17T12:22:51.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:50 smithi016 ceph-mon[34212]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:22:51.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:50 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3212434883"}]': finished 2024-09-17T12:22:51.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:50 smithi016 ceph-mon[34212]: osdmap e86: 8 total, 8 up, 8 in 2024-09-17T12:22:51.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:50 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3432043133' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/29394240"}]: dispatch 2024-09-17T12:22:51.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:50 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/29394240"}]: dispatch 2024-09-17T12:22:52.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:51 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/29394240"}]': finished 2024-09-17T12:22:52.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:51 smithi196 ceph-mon[27428]: osdmap e87: 8 total, 8 up, 8 in 2024-09-17T12:22:52.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:51 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1447195585' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4014507619"}]: dispatch 2024-09-17T12:22:52.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:51 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:22:52.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:51 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-09-17T12:22:52.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:51 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-09-17T12:22:52.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:51 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-09-17T12:22:52.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:51 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-09-17T12:22:52.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:51 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-17T12:22:52.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:51 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-09-17T12:22:52.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:51 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-09-17T12:22:52.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:51 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-17T12:22:52.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:51 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:22:52.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/29394240"}]': finished 2024-09-17T12:22:52.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[34212]: osdmap e87: 8 total, 8 up, 8 in 2024-09-17T12:22:52.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1447195585' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4014507619"}]: dispatch 2024-09-17T12:22:52.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:22:52.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-09-17T12:22:52.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-09-17T12:22:52.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-09-17T12:22:52.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-09-17T12:22:52.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-17T12:22:52.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-09-17T12:22:52.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-09-17T12:22:52.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-17T12:22:52.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:22:52.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/29394240"}]': finished 2024-09-17T12:22:52.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[23719]: osdmap e87: 8 total, 8 up, 8 in 2024-09-17T12:22:52.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1447195585' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4014507619"}]: dispatch 2024-09-17T12:22:52.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:22:52.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-09-17T12:22:52.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-09-17T12:22:52.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-09-17T12:22:52.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-09-17T12:22:52.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-17T12:22:52.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-09-17T12:22:52.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-09-17T12:22:52.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-17T12:22:52.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:51 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:22:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[23719]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1447195585' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4014507619"}]': finished 2024-09-17T12:22:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-09-17T12:22:53.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-09-17T12:22:53.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-17T12:22:53.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-17T12:22:53.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[23719]: osdmap e88: 8 total, 8 up, 8 in 2024-09-17T12:22:53.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3922502831' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3754043720"}]: dispatch 2024-09-17T12:22:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[34212]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1447195585' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4014507619"}]': finished 2024-09-17T12:22:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-09-17T12:22:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-09-17T12:22:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-17T12:22:53.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-17T12:22:53.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[34212]: osdmap e88: 8 total, 8 up, 8 in 2024-09-17T12:22:53.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:53 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3922502831' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3754043720"}]: dispatch 2024-09-17T12:22:53.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:53 smithi196 ceph-mon[27428]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:22:53.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:53 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1447195585' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4014507619"}]': finished 2024-09-17T12:22:53.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:53 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-09-17T12:22:53.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:53 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-09-17T12:22:53.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:53 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-17T12:22:53.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:53 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-17T12:22:53.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:53 smithi196 ceph-mon[27428]: osdmap e88: 8 total, 8 up, 8 in 2024-09-17T12:22:53.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:53 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3922502831' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3754043720"}]: dispatch 2024-09-17T12:22:54.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:54 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3922502831' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3754043720"}]': finished 2024-09-17T12:22:54.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:54 smithi016 ceph-mon[34212]: osdmap e89: 8 total, 8 up, 8 in 2024-09-17T12:22:54.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:54 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/103875208' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/218077182"}]: dispatch 2024-09-17T12:22:54.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:54 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/218077182"}]: dispatch 2024-09-17T12:22:54.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:54 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3922502831' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3754043720"}]': finished 2024-09-17T12:22:54.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:54 smithi016 ceph-mon[23719]: osdmap e89: 8 total, 8 up, 8 in 2024-09-17T12:22:54.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:54 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/103875208' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/218077182"}]: dispatch 2024-09-17T12:22:54.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:54 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/218077182"}]: dispatch 2024-09-17T12:22:54.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:54 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3922502831' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3754043720"}]': finished 2024-09-17T12:22:54.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:54 smithi196 ceph-mon[27428]: osdmap e89: 8 total, 8 up, 8 in 2024-09-17T12:22:54.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:54 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/103875208' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/218077182"}]: dispatch 2024-09-17T12:22:54.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:54 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/218077182"}]: dispatch 2024-09-17T12:22:55.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:55 smithi016 ceph-mon[23719]: pgmap v92: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 5 op/s 2024-09-17T12:22:55.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:55 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/218077182"}]': finished 2024-09-17T12:22:55.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:55 smithi016 ceph-mon[23719]: osdmap e90: 8 total, 8 up, 8 in 2024-09-17T12:22:55.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:55 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2298970912' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/610303924"}]: dispatch 2024-09-17T12:22:55.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:55 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/610303924"}]: dispatch 2024-09-17T12:22:55.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:55 smithi016 ceph-mon[34212]: pgmap v92: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 5 op/s 2024-09-17T12:22:55.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:55 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/218077182"}]': finished 2024-09-17T12:22:55.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:55 smithi016 ceph-mon[34212]: osdmap e90: 8 total, 8 up, 8 in 2024-09-17T12:22:55.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:55 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2298970912' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/610303924"}]: dispatch 2024-09-17T12:22:55.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:55 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/610303924"}]: dispatch 2024-09-17T12:22:55.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:55 smithi196 ceph-mon[27428]: pgmap v92: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 5 op/s 2024-09-17T12:22:55.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:55 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/218077182"}]': finished 2024-09-17T12:22:55.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:55 smithi196 ceph-mon[27428]: osdmap e90: 8 total, 8 up, 8 in 2024-09-17T12:22:55.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:55 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2298970912' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/610303924"}]: dispatch 2024-09-17T12:22:55.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:55 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/610303924"}]: dispatch 2024-09-17T12:22:56.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:56 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/610303924"}]': finished 2024-09-17T12:22:56.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:56 smithi016 ceph-mon[23719]: osdmap e91: 8 total, 8 up, 8 in 2024-09-17T12:22:56.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:56 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/764648111' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3785112609"}]: dispatch 2024-09-17T12:22:56.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:56 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3785112609"}]: dispatch 2024-09-17T12:22:56.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:22:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:22:56] "GET /metrics HTTP/1.1" 200 38345 "" "Prometheus/2.43.0" 2024-09-17T12:22:56.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:56 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/610303924"}]': finished 2024-09-17T12:22:56.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:56 smithi016 ceph-mon[34212]: osdmap e91: 8 total, 8 up, 8 in 2024-09-17T12:22:56.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:56 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/764648111' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3785112609"}]: dispatch 2024-09-17T12:22:56.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:56 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3785112609"}]: dispatch 2024-09-17T12:22:56.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:56 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/610303924"}]': finished 2024-09-17T12:22:56.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:56 smithi196 ceph-mon[27428]: osdmap e91: 8 total, 8 up, 8 in 2024-09-17T12:22:56.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:56 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/764648111' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3785112609"}]: dispatch 2024-09-17T12:22:56.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:56 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3785112609"}]: dispatch 2024-09-17T12:22:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:57 smithi016 ceph-mon[23719]: pgmap v95: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 5 op/s 2024-09-17T12:22:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:57 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3785112609"}]': finished 2024-09-17T12:22:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:57 smithi016 ceph-mon[23719]: osdmap e92: 8 total, 8 up, 8 in 2024-09-17T12:22:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:57 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3635418362' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4014507619"}]: dispatch 2024-09-17T12:22:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:57 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4014507619"}]: dispatch 2024-09-17T12:22:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:57 smithi016 ceph-mon[34212]: pgmap v95: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 5 op/s 2024-09-17T12:22:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:57 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3785112609"}]': finished 2024-09-17T12:22:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:57 smithi016 ceph-mon[34212]: osdmap e92: 8 total, 8 up, 8 in 2024-09-17T12:22:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:57 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3635418362' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4014507619"}]: dispatch 2024-09-17T12:22:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:57 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4014507619"}]: dispatch 2024-09-17T12:22:57.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:57 smithi196 ceph-mon[27428]: pgmap v95: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 5 op/s 2024-09-17T12:22:57.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:57 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3785112609"}]': finished 2024-09-17T12:22:57.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:57 smithi196 ceph-mon[27428]: osdmap e92: 8 total, 8 up, 8 in 2024-09-17T12:22:57.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:57 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3635418362' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4014507619"}]: dispatch 2024-09-17T12:22:57.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:57 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4014507619"}]: dispatch 2024-09-17T12:22:58.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:58 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4014507619"}]': finished 2024-09-17T12:22:58.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:58 smithi016 ceph-mon[34212]: osdmap e93: 8 total, 8 up, 8 in 2024-09-17T12:22:58.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:58 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3500621145' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3754043720"}]: dispatch 2024-09-17T12:22:58.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:58 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4014507619"}]': finished 2024-09-17T12:22:58.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:58 smithi016 ceph-mon[23719]: osdmap e93: 8 total, 8 up, 8 in 2024-09-17T12:22:58.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:58 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3500621145' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3754043720"}]: dispatch 2024-09-17T12:22:58.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:58 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4014507619"}]': finished 2024-09-17T12:22:58.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:58 smithi196 ceph-mon[27428]: osdmap e93: 8 total, 8 up, 8 in 2024-09-17T12:22:58.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:58 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3500621145' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3754043720"}]: dispatch 2024-09-17T12:22:59.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:59 smithi016 ceph-mon[23719]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:22:59.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:59 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3500621145' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3754043720"}]': finished 2024-09-17T12:22:59.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:22:59 smithi016 ceph-mon[23719]: osdmap e94: 8 total, 8 up, 8 in 2024-09-17T12:22:59.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:59 smithi016 ceph-mon[34212]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:22:59.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:59 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3500621145' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3754043720"}]': finished 2024-09-17T12:22:59.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:22:59 smithi016 ceph-mon[34212]: osdmap e94: 8 total, 8 up, 8 in 2024-09-17T12:22:59.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:59 smithi196 ceph-mon[27428]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:22:59.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:59 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3500621145' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3754043720"}]': finished 2024-09-17T12:22:59.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:22:59 smithi196 ceph-mon[27428]: osdmap e94: 8 total, 8 up, 8 in 2024-09-17T12:23:01.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:01 smithi016 ceph-mon[34212]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:23:01.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:01 smithi016 ceph-mon[23719]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:23:01.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:01 smithi196 ceph-mon[27428]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:23:02.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:02 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:02.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:02 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:02.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:02 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:03.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:03 smithi196 ceph-mon[27428]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:23:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:03 smithi016 ceph-mon[23719]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:23:03.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:03 smithi016 ceph-mon[34212]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:23:05.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:05 smithi016 ceph-mon[23719]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:23:05.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:05 smithi016 ceph-mon[34212]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:23:05.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:05 smithi196 ceph-mon[27428]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:23:06.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:23:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:23:06] "GET /metrics HTTP/1.1" 200 38345 "" "Prometheus/2.43.0" 2024-09-17T12:23:07.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:07 smithi016 ceph-mon[23719]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:23:07.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:07 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:23:07.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:07 smithi016 ceph-mon[34212]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:23:07.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:07 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:23:07.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:07 smithi196 ceph-mon[27428]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:23:07.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:07 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:23:09.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:09 smithi016 ceph-mon[23719]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:23:09.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:09 smithi016 ceph-mon[34212]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:23:09.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:09 smithi196 ceph-mon[27428]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:23:11.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:11 smithi016 ceph-mon[23719]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2024-09-17T12:23:11.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:11 smithi016 ceph-mon[34212]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2024-09-17T12:23:11.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:11 smithi196 ceph-mon[27428]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2024-09-17T12:23:12.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:12 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:12.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:12 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:12.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:12 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:13.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:13 smithi196 ceph-mon[27428]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:13.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:13 smithi016 ceph-mon[23719]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:13.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:13 smithi016 ceph-mon[34212]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:15.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:15 smithi196 ceph-mon[27428]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:15.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:15 smithi016 ceph-mon[23719]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:15.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:15 smithi016 ceph-mon[34212]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:16.593 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:23:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:23:16] "GET /metrics HTTP/1.1" 200 38358 "" "Prometheus/2.43.0" 2024-09-17T12:23:17.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:17 smithi196 ceph-mon[27428]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:17.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:17 smithi016 ceph-mon[23719]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:17.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:17 smithi016 ceph-mon[34212]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:19 smithi196 ceph-mon[27428]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:19.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:19 smithi016 ceph-mon[23719]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:19.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:19 smithi016 ceph-mon[34212]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:21.481 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:21 smithi016 ceph-mon[23719]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:21.482 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:21 smithi016 ceph-mon[34212]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:21.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:21 smithi196 ceph-mon[27428]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:22.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:22 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:22.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:22 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:23:22.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:22 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:22.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:22 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:23:22.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:22 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:22.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:22 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:23:23.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:23 smithi196 ceph-mon[27428]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:23.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:23 smithi016 ceph-mon[23719]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:23.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:23 smithi016 ceph-mon[34212]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:25.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:25 smithi196 ceph-mon[27428]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:25.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:25 smithi016 ceph-mon[23719]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:25.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:25 smithi016 ceph-mon[34212]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:26.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:23:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:23:26] "GET /metrics HTTP/1.1" 200 38362 "" "Prometheus/2.43.0" 2024-09-17T12:23:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:27 smithi196 ceph-mon[27428]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:27.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:27 smithi016 ceph-mon[23719]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:27.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:27 smithi016 ceph-mon[34212]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:29.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:29 smithi196 ceph-mon[27428]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:29.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:29 smithi016 ceph-mon[23719]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:29.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:29 smithi016 ceph-mon[34212]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:31.492 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:31 smithi016 ceph-mon[23719]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:31.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:31 smithi016 ceph-mon[34212]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:31.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:31 smithi196 ceph-mon[27428]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:32.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:32 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:32.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:32 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:32.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:32 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:33.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:33 smithi196 ceph-mon[27428]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:33.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:33 smithi016 ceph-mon[23719]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:33.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:33 smithi016 ceph-mon[34212]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:35.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:35 smithi196 ceph-mon[27428]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:35.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:35 smithi016 ceph-mon[23719]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:35.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:35 smithi016 ceph-mon[34212]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:36.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:23:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:23:36] "GET /metrics HTTP/1.1" 200 38362 "" "Prometheus/2.43.0" 2024-09-17T12:23:37.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:37 smithi196 ceph-mon[27428]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:37.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:37 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:23:37.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:37 smithi016 ceph-mon[23719]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:37.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:37 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:23:37.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:37 smithi016 ceph-mon[34212]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:37.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:37 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:23:39.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:39 smithi196 ceph-mon[27428]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:39.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:39 smithi016 ceph-mon[23719]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:39.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:39 smithi016 ceph-mon[34212]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:41.503 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:41 smithi016 ceph-mon[23719]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:41.503 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:41 smithi016 ceph-mon[34212]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:41.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:41 smithi196 ceph-mon[27428]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:42.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:42 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:42.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:42 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:42.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:42 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:43.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:43 smithi196 ceph-mon[27428]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:43.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:43 smithi016 ceph-mon[23719]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:43.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:43 smithi016 ceph-mon[34212]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:45.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:45 smithi196 ceph-mon[27428]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:45.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:45 smithi016 ceph-mon[23719]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:45.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:45 smithi016 ceph-mon[34212]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:46.592 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:23:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:23:46] "GET /metrics HTTP/1.1" 200 38356 "" "Prometheus/2.43.0" 2024-09-17T12:23:47.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:47 smithi196 ceph-mon[27428]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:47.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:47 smithi016 ceph-mon[23719]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:47.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:47 smithi016 ceph-mon[34212]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:49.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:49 smithi196 ceph-mon[27428]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:49.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:49 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:23:49.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:49 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:23:49.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:49 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:23:49.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:49 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:23:49.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:49 smithi016 ceph-mon[23719]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:49.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:49 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:23:49.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:49 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:23:49.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:49 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:23:49.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:49 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:23:49.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:49 smithi016 ceph-mon[34212]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:49.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:49 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:23:49.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:49 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:23:49.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:49 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:23:49.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:49 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:23:51.513 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:51 smithi016 ceph-mon[23719]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:51.514 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:51 smithi016 ceph-mon[34212]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:51.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:51 smithi196 ceph-mon[27428]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:52.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:52 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:52.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:23:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:52 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:52.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:23:52.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:52 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:23:52.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:23:53.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:53 smithi196 ceph-mon[27428]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:53.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:53 smithi016 ceph-mon[23719]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:53.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:53 smithi016 ceph-mon[34212]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:55.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:55 smithi196 ceph-mon[27428]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:55.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:55 smithi016 ceph-mon[23719]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:55.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:55 smithi016 ceph-mon[34212]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:56.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:23:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:23:56] "GET /metrics HTTP/1.1" 200 38357 "" "Prometheus/2.43.0" 2024-09-17T12:23:57.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:57 smithi196 ceph-mon[27428]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:57.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:57 smithi016 ceph-mon[23719]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:57.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:57 smithi016 ceph-mon[34212]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:23:59.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:23:59 smithi196 ceph-mon[27428]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:59.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:23:59 smithi016 ceph-mon[23719]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:23:59.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:23:59 smithi016 ceph-mon[34212]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:01.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:01 smithi196 ceph-mon[27428]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:01.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:01 smithi016 ceph-mon[23719]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:01.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:01 smithi016 ceph-mon[34212]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:02.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:02 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:02.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:02 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:02.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:02 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:03.678 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:03 smithi196 ceph-mon[27428]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:03.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:03 smithi016 ceph-mon[23719]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:03.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:03 smithi016 ceph-mon[34212]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:05 smithi016 ceph-mon[23719]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:05.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:05 smithi016 ceph-mon[34212]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:05.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:05 smithi196 ceph-mon[27428]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:06.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:24:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:24:06] "GET /metrics HTTP/1.1" 200 38357 "" "Prometheus/2.43.0" 2024-09-17T12:24:07.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:07 smithi016 ceph-mon[23719]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:07.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:07 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:24:07.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:07 smithi016 ceph-mon[34212]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:07.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:07 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:24:07.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:07 smithi196 ceph-mon[27428]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:07.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:07 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:24:09.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:09 smithi016 ceph-mon[23719]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:09.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:09 smithi016 ceph-mon[34212]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:09.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:09 smithi196 ceph-mon[27428]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:11.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:11 smithi016 ceph-mon[23719]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:11.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:11 smithi016 ceph-mon[34212]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:11.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:11 smithi196 ceph-mon[27428]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:12.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:12 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:12.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:12 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:12.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:12 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:13.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:13 smithi016 ceph-mon[23719]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:13.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:13 smithi016 ceph-mon[34212]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:13.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:13 smithi196 ceph-mon[27428]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:15.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:15 smithi016 ceph-mon[23719]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:15.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:15 smithi016 ceph-mon[34212]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:15.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:15 smithi196 ceph-mon[27428]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:16.593 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:24:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:24:16] "GET /metrics HTTP/1.1" 200 38358 "" "Prometheus/2.43.0" 2024-09-17T12:24:17.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:17 smithi016 ceph-mon[23719]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:17.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:17 smithi016 ceph-mon[34212]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:17.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:17 smithi196 ceph-mon[27428]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:19.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:19 smithi016 ceph-mon[23719]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:19.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:19 smithi016 ceph-mon[34212]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:19.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:19 smithi196 ceph-mon[27428]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:21.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:21 smithi016 ceph-mon[23719]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:21.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:21 smithi016 ceph-mon[34212]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:21.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:21 smithi196 ceph-mon[27428]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:22.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:22 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:22.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:22 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:24:22.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:22 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:22.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:22 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:24:22.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:22 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:22.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:22 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:24:23.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:23 smithi016 ceph-mon[23719]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:23.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:23 smithi016 ceph-mon[34212]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:23.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:23 smithi196 ceph-mon[27428]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:25.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:25 smithi016 ceph-mon[23719]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:25.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:25 smithi016 ceph-mon[34212]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:25.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:25 smithi196 ceph-mon[27428]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:26.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:24:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:24:26] "GET /metrics HTTP/1.1" 200 38359 "" "Prometheus/2.43.0" 2024-09-17T12:24:27.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:27 smithi016 ceph-mon[23719]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:27.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:27 smithi016 ceph-mon[34212]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:27.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:27 smithi196 ceph-mon[27428]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:29.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:29 smithi016 ceph-mon[23719]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:29.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:29 smithi016 ceph-mon[34212]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:29.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:29 smithi196 ceph-mon[27428]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:31 smithi016 ceph-mon[23719]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:31.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:31 smithi016 ceph-mon[34212]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:31.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:31 smithi196 ceph-mon[27428]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:32.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:32 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:32.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:32 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:32.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:32 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:33.678 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:33 smithi196 ceph-mon[27428]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:33.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:33 smithi016 ceph-mon[23719]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:33.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:33 smithi016 ceph-mon[34212]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:35.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:35 smithi016 ceph-mon[23719]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:35.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:35 smithi016 ceph-mon[34212]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:35.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:35 smithi196 ceph-mon[27428]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:36.320 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:24:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:24:36] "GET /metrics HTTP/1.1" 200 38359 "" "Prometheus/2.43.0" 2024-09-17T12:24:36.674 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-17T12:24:36.895 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:24:37.653 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:37 smithi016 ceph-mon[23719]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:37.653 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:37 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:24:37.653 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:37 smithi016 ceph-mon[34212]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:37.654 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:37 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:24:37.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:37 smithi196 ceph-mon[27428]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:37 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:24:39.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-17T12:24:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:24:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:39 smithi016 ceph-mon[23719]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:39.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:39 smithi016 ceph-mon[34212]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:39.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:39 smithi196 ceph-mon[27428]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:41.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:41 smithi016 ceph-mon[23719]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:41.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:41 smithi016 ceph-mon[34212]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:41.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:41 smithi196 ceph-mon[27428]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:41.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-17T12:24:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:24:42.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:42 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:42.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:42 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:42.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:42 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:43.682 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:43 smithi016 ceph-mon[23719]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:43.682 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:43 smithi016 ceph-mon[34212]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:43.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:43 smithi196 ceph-mon[27428]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:44.234 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-17T12:24:44.444 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:24:45.619 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:45 smithi016 ceph-mon[23719]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:45.619 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:45 smithi016 ceph-mon[34212]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:45.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:45 smithi196 ceph-mon[27428]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:46.170 INFO:teuthology.orchestra.run.smithi016.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169 2024-09-17T12:24:46.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:24:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:24:46] "GET /metrics HTTP/1.1" 200 38358 "" "Prometheus/2.43.0" 2024-09-17T12:24:46.756 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T12:24:46.770 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-09-17T12:24:46.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-09-17T12:24:46.991 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:24:47.422 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[23719]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:47.422 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[23719]: from='client.24932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:47.422 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[23719]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169 2024-09-17T12:24:47.423 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:24:47.423 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:24:47.423 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:24:47.423 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:24:47.423 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:24:47.423 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[23719]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169 2024-09-17T12:24:47.423 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[34212]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:47.423 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[34212]: from='client.24932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:47.423 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[34212]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169 2024-09-17T12:24:47.423 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:24:47.424 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:24:47.424 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:24:47.424 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:24:47.424 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:24:47.424 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:47 smithi016 ceph-mon[34212]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169 2024-09-17T12:24:47.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:47 smithi196 ceph-mon[27428]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:47.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:47 smithi196 ceph-mon[27428]: from='client.24932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:47.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:47 smithi196 ceph-mon[27428]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169 2024-09-17T12:24:47.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:47 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:24:47.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:47 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:24:47.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:47 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:24:47.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:47 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:24:47.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:47 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:24:47.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:47 smithi196 ceph-mon[27428]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169 2024-09-17T12:24:48.893 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:24:49.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:49 smithi016 ceph-mon[23719]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:49.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:49 smithi016 ceph-mon[34212]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:24:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (2m) 2m ago 3m 24.2M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:24:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (3m) 2m ago 3m 83.3M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:24:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (2m) 2m ago 2m 83.0M - 3.8 980a0bfcd684 bc9ab4f22b6b 2024-09-17T12:24:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,8765 running (7m) 2m ago 7m 438M - 18.2.4-855-g19e2a2be 980a0bfcd684 842e33f5d7dd 2024-09-17T12:24:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:9283,8765 running (9m) 2m ago 9m 489M - 18.2.4-855-g19e2a2be 980a0bfcd684 caffea02f791 2024-09-17T12:24:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (9m) 2m ago 9m 50.4M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 5d21f110d6fc 2024-09-17T12:24:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (8m) 2m ago 8m 38.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 262ef622777e 2024-09-17T12:24:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (7m) 2m ago 7m 45.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 950b72634364 2024-09-17T12:24:49.523 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (3m) 2m ago 3m 15.3M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:24:49.523 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (3m) 2m ago 3m 8074k - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:24:49.523 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (7m) 2m ago 7m 58.7M 4017M 18.2.4-855-g19e2a2be 980a0bfcd684 e30aaffd3b5f 2024-09-17T12:24:49.523 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (6m) 2m ago 6m 50.9M 4017M 18.2.4-855-g19e2a2be 980a0bfcd684 3ad94c30c9c9 2024-09-17T12:24:49.523 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (6m) 2m ago 6m 51.9M 4017M 18.2.4-855-g19e2a2be 980a0bfcd684 1fa310695876 2024-09-17T12:24:49.523 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (6m) 2m ago 6m 56.2M 4017M 18.2.4-855-g19e2a2be 980a0bfcd684 4a4f8eeaff13 2024-09-17T12:24:49.524 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (5m) 2m ago 5m 50.9M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 afa20056b398 2024-09-17T12:24:49.524 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (5m) 2m ago 5m 46.2M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 f0dcabb8902f 2024-09-17T12:24:49.524 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (4m) 2m ago 4m 55.0M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:24:49.524 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (4m) 2m ago 4m 57.7M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:24:49.524 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (2m) 2m ago 4m 33.8M - 2.43.0 a07b618ecd1d cb4f31b0d350 2024-09-17T12:24:49.525 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (2m) 2m ago 2m 75.5M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:24:49.525 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (2m) 2m ago 2m 74.9M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:24:49.525 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (2m) 2m ago 2m 77.2M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:24:49.525 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (2m) 2m ago 2m 74.0M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:24:49.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:49 smithi196 ceph-mon[27428]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:49.884 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:24:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:24:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T12:24:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:24:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:24:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T12:24:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:24:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:24:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T12:24:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:24:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:24:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:24:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:24:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:24:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 17 2024-09-17T12:24:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:24:49.885 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:24:50.193 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:24:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169", 2024-09-17T12:24:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:24:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:24:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-09-17T12:24:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "", 2024-09-17T12:24:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169 image", 2024-09-17T12:24:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:24:50.193 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:24:50.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:50 smithi016 ceph-mon[23719]: from='client.24938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:50.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:50 smithi016 ceph-mon[23719]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:50.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:50 smithi016 ceph-mon[23719]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:50.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:50 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3111547698' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:24:50.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:50 smithi016 ceph-mon[34212]: from='client.24938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:50.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:50 smithi016 ceph-mon[34212]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:50.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:50 smithi016 ceph-mon[34212]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:50.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:50 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3111547698' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:24:50.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:50 smithi196 ceph-mon[27428]: from='client.24938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:50.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:50 smithi196 ceph-mon[27428]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:50.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:50 smithi196 ceph-mon[27428]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:50.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:50 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3111547698' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:24:50.559 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-09-17T12:24:51.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:51 smithi016 ceph-mon[23719]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:51.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:51 smithi016 ceph-mon[23719]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:51.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:51 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/813062669' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:24:51.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:51 smithi016 ceph-mon[34212]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:51.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:51 smithi016 ceph-mon[34212]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:51.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:51 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/813062669' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:24:51.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:51 smithi196 ceph-mon[27428]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:51.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:51 smithi196 ceph-mon[27428]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:24:51.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:51 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/813062669' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:24:52.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:52 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:52.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:52 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:24:52.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:52 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:52.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:52 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:24:52.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:52 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:24:52.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:52 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:24:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:53 smithi016 ceph-mon[23719]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:53.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:53 smithi016 ceph-mon[34212]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:53.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:53 smithi196 ceph-mon[27428]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:55.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:55 smithi016 ceph-mon[23719]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:55.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:55 smithi016 ceph-mon[34212]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:55.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:55 smithi196 ceph-mon[27428]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:56.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:24:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:24:56] "GET /metrics HTTP/1.1" 200 38356 "" "Prometheus/2.43.0" 2024-09-17T12:24:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:57 smithi016 ceph-mon[23719]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:57.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:57 smithi016 ceph-mon[34212]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:57.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:57 smithi196 ceph-mon[27428]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:24:59.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:24:59 smithi016 ceph-mon[23719]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:59.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:24:59 smithi016 ceph-mon[34212]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:24:59.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:24:59 smithi196 ceph-mon[27428]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:01.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:01 smithi196 ceph-mon[27428]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:01.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:01 smithi016 ceph-mon[23719]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:01.582 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:01 smithi016 ceph-mon[34212]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:02.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:02 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:02.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:02 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:02.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:02 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:03.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:03 smithi196 ceph-mon[27428]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:03.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:03 smithi016 ceph-mon[23719]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:03.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:03 smithi016 ceph-mon[34212]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:05 smithi196 ceph-mon[27428]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:05.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:05 smithi016 ceph-mon[23719]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:05.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:05 smithi016 ceph-mon[34212]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:06.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:25:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:25:06] "GET /metrics HTTP/1.1" 200 38356 "" "Prometheus/2.43.0" 2024-09-17T12:25:07.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:07 smithi196 ceph-mon[27428]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:07.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:07 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:25:07.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:07 smithi016 ceph-mon[23719]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:07.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:07 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:25:07.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:07 smithi016 ceph-mon[34212]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:07.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:07 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:25:09.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:09 smithi196 ceph-mon[27428]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:09.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:09 smithi016 ceph-mon[23719]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:09.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:09 smithi016 ceph-mon[34212]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:11.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:11 smithi196 ceph-mon[27428]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:11.592 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:11 smithi016 ceph-mon[23719]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:11.592 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:11 smithi016 ceph-mon[34212]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:12.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:12 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:12.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:12 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:12.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:12 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:13.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:13 smithi196 ceph-mon[27428]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:13.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:13 smithi016 ceph-mon[34212]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:13.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:13 smithi016 ceph-mon[23719]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:15.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:15 smithi196 ceph-mon[27428]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:15.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:15 smithi016 ceph-mon[23719]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:15.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:15 smithi016 ceph-mon[34212]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:16.592 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:25:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:25:16] "GET /metrics HTTP/1.1" 200 38356 "" "Prometheus/2.43.0" 2024-09-17T12:25:17.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:17 smithi196 ceph-mon[27428]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:17.602 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:17 smithi016 ceph-mon[34212]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:17.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:17 smithi016 ceph-mon[23719]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:19 smithi196 ceph-mon[27428]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:19.596 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:19 smithi016 ceph-mon[23719]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:19.596 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:19 smithi016 ceph-mon[34212]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:20.869 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:25:20.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:20 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:20.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:20 smithi016 ceph-mon[23719]: Upgrade: Target is version 19.3.0-4937-g74984932 (unknown) 2024-09-17T12:25:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:20 smithi016 ceph-mon[23719]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, digests ['quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb'] 2024-09-17T12:25:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:20 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:25:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:20 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:20 smithi016 ceph-mon[23719]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T12:25:20.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:20 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:20.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:20 smithi016 ceph-mon[34212]: Upgrade: Target is version 19.3.0-4937-g74984932 (unknown) 2024-09-17T12:25:20.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:20 smithi016 ceph-mon[34212]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, digests ['quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb'] 2024-09-17T12:25:20.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:20 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:25:20.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:20 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:20.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:20 smithi016 ceph-mon[34212]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T12:25:21.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:20 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:21.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:20 smithi196 ceph-mon[27428]: Upgrade: Target is version 19.3.0-4937-g74984932 (unknown) 2024-09-17T12:25:21.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:20 smithi196 ceph-mon[27428]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, digests ['quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb'] 2024-09-17T12:25:21.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:20 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:25:21.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:20 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:21.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:20 smithi196 ceph-mon[27428]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T12:25:21.428 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:25:21.428 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (3m) 2m ago 4m 24.2M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:25:21.428 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (4m) 2m ago 4m 83.3M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:25:21.428 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (2m) 2m ago 2m 83.0M - 3.8 980a0bfcd684 bc9ab4f22b6b 2024-09-17T12:25:21.428 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,8765 running (8m) 2m ago 8m 438M - 18.2.4-855-g19e2a2be 980a0bfcd684 842e33f5d7dd 2024-09-17T12:25:21.428 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:9283,8765 running (9m) 2m ago 9m 489M - 18.2.4-855-g19e2a2be 980a0bfcd684 caffea02f791 2024-09-17T12:25:21.428 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (9m) 2m ago 9m 50.4M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 5d21f110d6fc 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (8m) 2m ago 8m 38.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 262ef622777e 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (8m) 2m ago 8m 45.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 950b72634364 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (3m) 2m ago 3m 15.3M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (3m) 2m ago 3m 8074k - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (7m) 2m ago 7m 58.7M 4017M 18.2.4-855-g19e2a2be 980a0bfcd684 e30aaffd3b5f 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (7m) 2m ago 7m 50.9M 4017M 18.2.4-855-g19e2a2be 980a0bfcd684 3ad94c30c9c9 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (7m) 2m ago 7m 51.9M 4017M 18.2.4-855-g19e2a2be 980a0bfcd684 1fa310695876 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (6m) 2m ago 6m 56.2M 4017M 18.2.4-855-g19e2a2be 980a0bfcd684 4a4f8eeaff13 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (6m) 2m ago 6m 50.9M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 afa20056b398 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (5m) 2m ago 5m 46.2M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 f0dcabb8902f 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (5m) 2m ago 5m 55.0M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (5m) 2m ago 5m 57.7M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (3m) 2m ago 4m 33.8M - 2.43.0 a07b618ecd1d cb4f31b0d350 2024-09-17T12:25:21.429 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (2m) 2m ago 2m 75.5M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:25:21.430 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (2m) 2m ago 2m 74.9M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:25:21.430 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (2m) 2m ago 2m 77.2M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:25:21.430 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (2m) 2m ago 2m 74.0M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:25:21.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:21 smithi016 ceph-mon[23719]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:21.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:21 smithi016 ceph-mon[23719]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb on smithi196 2024-09-17T12:25:21.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:21 smithi016 ceph-mon[34212]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:21.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:21 smithi016 ceph-mon[34212]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb on smithi196 2024-09-17T12:25:21.761 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:25:21.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:25:21.761 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T12:25:21.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:25:21.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:25:21.761 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T12:25:21.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:25:21.762 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:25:21.762 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T12:25:21.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:25:21.762 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:25:21.762 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:25:21.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:25:21.762 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:25:21.762 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 17 2024-09-17T12:25:21.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:25:21.762 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:25:22.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:21 smithi196 ceph-mon[27428]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:22.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:21 smithi196 ceph-mon[27428]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb on smithi196 2024-09-17T12:25:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:25:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:25:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:25:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:25:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-09-17T12:25:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "0/23 daemons upgraded", 2024-09-17T12:25:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb image on host smithi196", 2024-09-17T12:25:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:25:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:25:22.374 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-09-17T12:25:22.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[34212]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:22.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[34212]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:22.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[34212]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:22.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:22.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:25:22.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1645533407' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:25:22.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3798214466' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:25:22.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[23719]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:22.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[23719]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:22.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[23719]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:22.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:22.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:25:22.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1645533407' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:25:22.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:22 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3798214466' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:25:23.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:22 smithi196 ceph-mon[27428]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:22 smithi196 ceph-mon[27428]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:22 smithi196 ceph-mon[27428]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:22 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:22 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:25:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:22 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1645533407' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:25:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:22 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3798214466' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:25:23.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:23 smithi016 ceph-mon[23719]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:23.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:23 smithi016 ceph-mon[23719]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:23.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:23 smithi016 ceph-mon[34212]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:23.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:23 smithi016 ceph-mon[34212]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:24.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:23 smithi196 ceph-mon[27428]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:24.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:23 smithi196 ceph-mon[27428]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:24 smithi016 ceph-mon[23719]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:24.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:24 smithi016 ceph-mon[34212]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:25.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:24 smithi196 ceph-mon[27428]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:26.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:25:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:25:26] "GET /metrics HTTP/1.1" 200 38355 "" "Prometheus/2.43.0" 2024-09-17T12:25:27.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:26 smithi016 ceph-mon[23719]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:27.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:26 smithi016 ceph-mon[34212]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:27.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:26 smithi196 ceph-mon[27428]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:29.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:28 smithi016 ceph-mon[23719]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:29.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:28 smithi016 ceph-mon[34212]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:29.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:28 smithi196 ceph-mon[27428]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:31.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:30 smithi016 ceph-mon[23719]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:31.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:30 smithi016 ceph-mon[34212]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:31.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:30 smithi196 ceph-mon[27428]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:31.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:31 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:31.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:31 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:32.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:31 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:33.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:32 smithi196 ceph-mon[27428]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:33.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:32 smithi016 ceph-mon[23719]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:33.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:32 smithi016 ceph-mon[34212]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:34 smithi016 ceph-mon[23719]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:35.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:34 smithi016 ceph-mon[34212]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:35.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:34 smithi196 ceph-mon[27428]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:36.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:25:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:25:36] "GET /metrics HTTP/1.1" 200 38355 "" "Prometheus/2.43.0" 2024-09-17T12:25:37.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:36 smithi016 ceph-mon[23719]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:37.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:36 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:25:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:36 smithi016 ceph-mon[34212]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:36 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:25:37.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:36 smithi196 ceph-mon[27428]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:37.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:36 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:25:39.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:38 smithi016 ceph-mon[23719]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:39.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:38 smithi016 ceph-mon[34212]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:39.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:38 smithi196 ceph-mon[27428]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:41.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:40 smithi016 ceph-mon[23719]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:41.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:40 smithi016 ceph-mon[34212]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:41.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:40 smithi196 ceph-mon[27428]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:41.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:41 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:41.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:41 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:42.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:41 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:43.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:42 smithi016 ceph-mon[23719]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:43.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:42 smithi016 ceph-mon[34212]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:43.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:42 smithi196 ceph-mon[27428]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:45.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:44 smithi016 ceph-mon[23719]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:45.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:44 smithi016 ceph-mon[34212]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:45.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:44 smithi196 ceph-mon[27428]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:46.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:25:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:25:46] "GET /metrics HTTP/1.1" 200 38355 "" "Prometheus/2.43.0" 2024-09-17T12:25:46.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:46 smithi016 ceph-mon[23719]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T12:25:46.949 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:46 smithi016 ceph-mon[34212]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T12:25:47.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:46 smithi196 ceph-mon[27428]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T12:25:49.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:48 smithi016 ceph-mon[23719]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:49.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:48 smithi016 ceph-mon[34212]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:49.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:48 smithi196 ceph-mon[27428]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:51.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:50 smithi016 ceph-mon[23719]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T12:25:51.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:50 smithi016 ceph-mon[34212]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T12:25:51.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:50 smithi196 ceph-mon[27428]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T12:25:51.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:51 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:51.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:51 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:25:51.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:51 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:51.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:51 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:25:52.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:51 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:25:52.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:51 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:25:52.684 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:25:52.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:52 smithi016 ceph-mon[23719]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T12:25:52.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:52 smithi016 ceph-mon[34212]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T12:25:53.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:52 smithi196 ceph-mon[27428]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T12:25:53.242 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:25:53.242 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (4m) 3m ago 4m 24.2M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:25:53.242 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (4m) 3m ago 4m 83.3M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:25:53.242 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (3m) 3m ago 3m 83.0M - 3.8 980a0bfcd684 bc9ab4f22b6b 2024-09-17T12:25:53.242 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,8765 running (8m) 3m ago 8m 438M - 18.2.4-855-g19e2a2be 980a0bfcd684 842e33f5d7dd 2024-09-17T12:25:53.242 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:9283,8765 running (10m) 3m ago 10m 489M - 18.2.4-855-g19e2a2be 980a0bfcd684 caffea02f791 2024-09-17T12:25:53.242 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (10m) 3m ago 10m 50.4M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 5d21f110d6fc 2024-09-17T12:25:53.242 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (9m) 3m ago 9m 38.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 262ef622777e 2024-09-17T12:25:53.242 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (8m) 3m ago 8m 45.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 950b72634364 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (4m) 3m ago 4m 15.3M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (4m) 3m ago 4m 8074k - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (8m) 3m ago 8m 58.7M 4017M 18.2.4-855-g19e2a2be 980a0bfcd684 e30aaffd3b5f 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (8m) 3m ago 8m 50.9M 4017M 18.2.4-855-g19e2a2be 980a0bfcd684 3ad94c30c9c9 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (7m) 3m ago 7m 51.9M 4017M 18.2.4-855-g19e2a2be 980a0bfcd684 1fa310695876 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (7m) 3m ago 7m 56.2M 4017M 18.2.4-855-g19e2a2be 980a0bfcd684 4a4f8eeaff13 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (6m) 3m ago 6m 50.9M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 afa20056b398 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (6m) 3m ago 6m 46.2M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 f0dcabb8902f 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (6m) 3m ago 6m 55.0M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (5m) 3m ago 5m 57.7M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (3m) 3m ago 5m 33.8M - 2.43.0 a07b618ecd1d cb4f31b0d350 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (3m) 3m ago 3m 75.5M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (3m) 3m ago 3m 74.9M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:25:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (3m) 3m ago 3m 77.2M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:25:53.244 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (3m) 3m ago 3m 74.0M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:25:53.574 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:25:53.574 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:25:53.574 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T12:25:53.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:25:53.575 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:25:53.575 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T12:25:53.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:25:53.575 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:25:53.575 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T12:25:53.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:25:53.575 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:25:53.575 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:25:53.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:25:53.575 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:25:53.575 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 17 2024-09-17T12:25:53.575 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:25:53.576 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:25:53.860 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:25:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:25:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:25:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:25:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-09-17T12:25:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "0/23 daemons upgraded", 2024-09-17T12:25:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb image on host smithi196", 2024-09-17T12:25:53.861 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:25:53.861 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:25:54.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:53 smithi016 ceph-mon[23719]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:54.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:53 smithi016 ceph-mon[23719]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:54.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:53 smithi016 ceph-mon[23719]: from='client.25022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:54.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:53 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/350841834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:25:54.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:53 smithi016 ceph-mon[34212]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:54.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:53 smithi016 ceph-mon[34212]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:54.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:53 smithi016 ceph-mon[34212]: from='client.25022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:54.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:53 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/350841834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:25:54.192 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-09-17T12:25:54.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:53 smithi196 ceph-mon[27428]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:54.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:53 smithi196 ceph-mon[27428]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:54.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:53 smithi196 ceph-mon[27428]: from='client.25022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:54.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:53 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/350841834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:25:54.935 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:54 smithi196 ceph-mon[27428]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:54.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:54 smithi196 ceph-mon[27428]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:54.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:54 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/54240254' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:25:54.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:54 smithi196 ceph-mon[27428]: Upgrade: Updating mgr.x 2024-09-17T12:25:54.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:54.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:54 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:25:54.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:54 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:25:54.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:54 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:25:54.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:54 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:25:54.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:54 smithi196 ceph-mon[27428]: Deploying daemon mgr.x on smithi196 2024-09-17T12:25:55.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[23719]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:55.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[23719]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/54240254' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:25:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[23719]: Upgrade: Updating mgr.x 2024-09-17T12:25:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:25:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:25:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:25:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:25:55.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[23719]: Deploying daemon mgr.x on smithi196 2024-09-17T12:25:55.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[34212]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:55.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[34212]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:25:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/54240254' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:25:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[34212]: Upgrade: Updating mgr.x 2024-09-17T12:25:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:25:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:25:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:25:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:25:55.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:54 smithi016 ceph-mon[34212]: Deploying daemon mgr.x on smithi196 2024-09-17T12:25:56.158 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:55 smithi196 systemd[1]: Stopping Ceph mgr.x for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:25:56.526 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:56 smithi196 podman[62265]: 2024-09-17 12:25:56.156256252 +0000 UTC m=+0.142683788 container died 842e33f5d7ddbee5f4bef909c44207f5245ce547a3d55913665c74ca752da746 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-17T12:25:56.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:25:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:25:56] "GET /metrics HTTP/1.1" 200 38357 "" "Prometheus/2.43.0" 2024-09-17T12:25:56.802 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:56 smithi196 podman[62265]: 2024-09-17 12:25:56.53343606 +0000 UTC m=+0.519863642 container cleanup 842e33f5d7ddbee5f4bef909c44207f5245ce547a3d55913665c74ca752da746 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:25:56.802 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:56 smithi196 bash[62265]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x 2024-09-17T12:25:57.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:56 smithi016 ceph-mon[23719]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:57.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:56 smithi016 ceph-mon[34212]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:57.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:56 smithi196 ceph-mon[27428]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:25:57.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:56 smithi196 podman[62277]: 2024-09-17 12:25:56.801766746 +0000 UTC m=+0.648955978 container remove 842e33f5d7ddbee5f4bef909c44207f5245ce547a3d55913665c74ca752da746 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-17T12:25:57.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:56 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-17T12:25:57.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:56 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-17T12:25:57.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:56 smithi196 systemd[1]: Stopped Ceph mgr.x for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:25:57.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:56 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.x.service: Consumed 11.018s CPU time. 2024-09-17T12:25:57.694 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:57 smithi196 systemd[1]: Starting Ceph mgr.x for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:25:57.694 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:57 smithi196 podman[62406]: 2024-09-17 12:25:57.475218782 +0000 UTC m=+0.016053491 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:25:57.694 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:57 smithi196 podman[62406]: 2024-09-17 12:25:57.587182064 +0000 UTC m=+0.128016772 container create 7ba30e0f95c9f92915e51f5688cb04efc27d73a8be8509f36ed09d565fd0d3c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=main-7498493, ceph=True) 2024-09-17T12:25:58.026 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:57 smithi196 podman[62406]: 2024-09-17 12:25:57.814804385 +0000 UTC m=+0.355639111 container init 7ba30e0f95c9f92915e51f5688cb04efc27d73a8be8509f36ed09d565fd0d3c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-7498493) 2024-09-17T12:25:58.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:57 smithi196 podman[62406]: 2024-09-17 12:25:57.817876479 +0000 UTC m=+0.358711197 container start 7ba30e0f95c9f92915e51f5688cb04efc27d73a8be8509f36ed09d565fd0d3c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-7498493, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-17T12:25:58.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:57 smithi196 ceph-mgr[62425]: -- 172.21.15.196:0/824369181 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fb706dd0e0 con 0x55fb706bb400 2024-09-17T12:25:58.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:57 smithi196 bash[62406]: 7ba30e0f95c9f92915e51f5688cb04efc27d73a8be8509f36ed09d565fd0d3c7 2024-09-17T12:25:58.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:25:57 smithi196 systemd[1]: Started Ceph mgr.x for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:25:59.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:58 smithi196 ceph-mon[27428]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:59.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:58 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:59.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:58 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:59.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:25:58 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:25:59.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:58 smithi016 ceph-mon[23719]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:59.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:58 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:59.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:58 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:59.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:25:58 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:25:59.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:58 smithi016 ceph-mon[34212]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:25:59.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:58 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:59.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:58 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:25:59.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:25:58 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:01.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:00 smithi196 ceph-mon[27428]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:01.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:00 smithi016 ceph-mon[23719]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:01.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:00 smithi016 ceph-mon[34212]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:03.527 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: [17/Sep/2024:12:26:03] ENGINE Bus STARTING 2024-09-17T12:26:03.527 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: CherryPy Checker: 2024-09-17T12:26:03.527 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: The Application mounted at '' has an empty config. 2024-09-17T12:26:03.527 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: 2024-09-17T12:26:03.527 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: [17/Sep/2024:12:26:03] ENGINE Serving on http://:::9283 2024-09-17T12:26:03.527 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: [17/Sep/2024:12:26:03] ENGINE Bus STARTED 2024-09-17T12:26:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:26:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-mon[27428]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-mon[27428]: Standby manager daemon x restarted 2024-09-17T12:26:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-mon[27428]: Standby manager daemon x started 2024-09-17T12:26:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:26:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:26:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:26:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:03 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:26:03.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:26:03.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[34212]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:03.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:03.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:03.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:03.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:03.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[34212]: Standby manager daemon x restarted 2024-09-17T12:26:03.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[34212]: Standby manager daemon x started 2024-09-17T12:26:03.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:26:03.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:26:03.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:26:03.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:26:03.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:26:03.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[23719]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:03.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:03.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:03.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:03.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:03.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[23719]: Standby manager daemon x restarted 2024-09-17T12:26:03.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[23719]: Standby manager daemon x started 2024-09-17T12:26:03.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:26:03.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:26:03.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:26:03.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:03 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:26:05.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:04 smithi196 ceph-mon[27428]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:05.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:04 smithi196 ceph-mon[27428]: mgrmap e23: y(active, since 5m), standbys: x 2024-09-17T12:26:05.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:04 smithi016 ceph-mon[23719]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:05.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:04 smithi016 ceph-mon[23719]: mgrmap e23: y(active, since 5m), standbys: x 2024-09-17T12:26:05.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:04 smithi016 ceph-mon[34212]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:05.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:04 smithi016 ceph-mon[34212]: mgrmap e23: y(active, since 5m), standbys: x 2024-09-17T12:26:06.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:26:06] "GET /metrics HTTP/1.1" 200 38357 "" "Prometheus/2.43.0" 2024-09-17T12:26:07.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:06 smithi016 ceph-mon[23719]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:07.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:06 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:07.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:06 smithi016 ceph-mon[34212]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:07.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:06 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:07.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:06 smithi196 ceph-mon[27428]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:07.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:06 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:08 smithi016 ceph-mon[23719]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:09.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:08 smithi016 ceph-mon[34212]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:09.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:08 smithi196 ceph-mon[27428]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:10 smithi016 ceph-mon[23719]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:11.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:10 smithi016 ceph-mon[34212]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:11.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:10 smithi196 ceph-mon[27428]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:13.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:12 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:26:13.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:12 smithi196 ceph-mon[27428]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:13.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:12 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:26:13.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:12 smithi016 ceph-mon[23719]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:13.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:12 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:26:13.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:12 smithi016 ceph-mon[34212]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:15.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:14 smithi196 ceph-mon[27428]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:15.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:14 smithi016 ceph-mon[23719]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:15.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:14 smithi016 ceph-mon[34212]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:16.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:26:16] "GET /metrics HTTP/1.1" 200 38350 "" "Prometheus/2.43.0" 2024-09-17T12:26:16.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:16.664+0000 7fbf6c9b3640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-17T12:26:16.776 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: [17/Sep/2024:12:26:16] ENGINE Bus STOPPING 2024-09-17T12:26:16.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ignoring --setuser ceph since I am not root 2024-09-17T12:26:16.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ignoring --setgroup ceph since I am not root 2024-09-17T12:26:16.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:16.812+0000 7fe04346f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T12:26:16.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:16.870+0000 7fe04346f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T12:26:17.276 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: [17/Sep/2024:12:26:16] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T12:26:17.276 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: [17/Sep/2024:12:26:16] ENGINE Bus STOPPED 2024-09-17T12:26:17.397 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:17.024+0000 7fe04346f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T12:26:17.397 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:17.095+0000 7fe04346f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T12:26:17.397 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:17.248+0000 7fe04346f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:26:17.397 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:17.394+0000 7fe04346f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T12:26:17.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:17.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:17.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:17.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:17.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:26:17.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:17.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:17.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T12:26:17.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T12:26:17.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: Activating manager daemon x 2024-09-17T12:26:17.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: osdmap e95: 8 total, 8 up, 8 in 2024-09-17T12:26:17.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T12:26:17.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: mgrmap e24: x(active, starting, since 0.0109421s) 2024-09-17T12:26:17.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:26:17.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:26:17.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:26:17.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:26:17.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:26:17.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:26:17.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:26:17.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:26:17.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:26:17.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:26:17.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:26:17.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:26:17.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:26:17.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:26:17.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:26:17.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[23719]: Manager daemon x is now available 2024-09-17T12:26:17.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:17.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:17.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:17.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:17.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:26:17.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:17.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:17.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T12:26:17.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T12:26:17.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: Activating manager daemon x 2024-09-17T12:26:17.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: osdmap e95: 8 total, 8 up, 8 in 2024-09-17T12:26:17.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T12:26:17.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: mgrmap e24: x(active, starting, since 0.0109421s) 2024-09-17T12:26:17.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:26:17.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:26:17.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:26:17.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:26:17.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:26:17.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:26:17.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:26:17.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:26:17.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:26:17.696 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:26:17.696 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:26:17.696 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:26:17.696 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:26:17.696 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:26:17.696 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:26:17.696 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-mon[34212]: Manager daemon x is now available 2024-09-17T12:26:17.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:26:17.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:17.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:17.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:17.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:26:17.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' 2024-09-17T12:26:17.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:17.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.14430 172.21.15.16:0/536375840' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T12:26:17.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T12:26:17.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: Activating manager daemon x 2024-09-17T12:26:17.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: osdmap e95: 8 total, 8 up, 8 in 2024-09-17T12:26:17.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.14430 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T12:26:17.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: mgrmap e24: x(active, starting, since 0.0109421s) 2024-09-17T12:26:17.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:26:17.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:26:17.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:26:17.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:26:17.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:26:17.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:26:17.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:26:17.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:26:17.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:26:17.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:26:17.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:26:17.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:26:17.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:26:17.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:26:17.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:26:17.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-mon[27428]: Manager daemon x is now available 2024-09-17T12:26:17.982 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:17.721+0000 7fe04346f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T12:26:17.982 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:17.791+0000 7fe04346f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T12:26:18.145 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: [17/Sep/2024:12:26:17] ENGINE Bus STARTING 2024-09-17T12:26:18.146 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: [17/Sep/2024:12:26:17] ENGINE Serving on http://:::9283 2024-09-17T12:26:18.146 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: [17/Sep/2024:12:26:17] ENGINE Bus STARTED 2024-09-17T12:26:18.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:17 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:17.984+0000 7fe04346f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T12:26:18.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: mgrmap e25: x(active, since 1.01059s) 2024-09-17T12:26:18.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-17T12:26:18.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-17T12:26:18.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-17T12:26:18.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: Checking for cert/key for grafana.a 2024-09-17T12:26:18.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: Migrating grafana.a cert to cert store 2024-09-17T12:26:18.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: Migrating grafana.a key to cert store 2024-09-17T12:26:18.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:18.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:18.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:26:18.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:26:18.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:26:18.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:26:18.922 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T12:26:18.922 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T12:26:18.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: mgrmap e25: x(active, since 1.01059s) 2024-09-17T12:26:18.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-17T12:26:18.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-17T12:26:18.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-17T12:26:18.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: Checking for cert/key for grafana.a 2024-09-17T12:26:18.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: Migrating grafana.a cert to cert store 2024-09-17T12:26:18.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: Migrating grafana.a key to cert store 2024-09-17T12:26:18.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:18.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:18.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:26:18.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:26:18.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:26:18.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:26:18.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T12:26:18.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T12:26:18.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: mgrmap e25: x(active, since 1.01059s) 2024-09-17T12:26:18.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-17T12:26:18.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-17T12:26:18.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-17T12:26:18.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: Checking for cert/key for grafana.a 2024-09-17T12:26:18.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: Migrating grafana.a cert to cert store 2024-09-17T12:26:18.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: Migrating grafana.a key to cert store 2024-09-17T12:26:18.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:18.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:18.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:18.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:26:18.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:26:18.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:26:18.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:26:18.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T12:26:18.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T12:26:19.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:18.917+0000 7fe04346f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T12:26:19.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:18 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:18.988+0000 7fe04346f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T12:26:19.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: 2024-09-17T12:26:18.961+0000 7fa94a130640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:12:26:18] ENGINE Error in HTTPServer.serve 2024-09-17T12:26:19.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Traceback (most recent call last): 2024-09-17T12:26:19.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:26:19.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: self._connections.run(self.expiration_interval) 2024-09-17T12:26:19.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:26:19.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: self._run(expiration_interval) 2024-09-17T12:26:19.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:26:19.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:26:19.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:26:19.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:26:19.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:26:19.278 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: s = self.context.wrap_socket( 2024-09-17T12:26:19.278 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:26:19.278 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: return self.sslsocket_class._create( 2024-09-17T12:26:19.278 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:26:19.278 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: self.do_handshake() 2024-09-17T12:26:19.278 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:26:19.278 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: self._sslobj.do_handshake() 2024-09-17T12:26:19.278 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:26:19.278 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: 2024-09-17T12:26:19.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:19 smithi016 ceph-mon[23719]: Deploying cephadm binary to smithi016 2024-09-17T12:26:19.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:19 smithi016 ceph-mon[23719]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:19.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:19 smithi016 ceph-mon[23719]: [17/Sep/2024:12:26:18] ENGINE Bus STARTING 2024-09-17T12:26:19.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:19 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:19.749+0000 7fe04346f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T12:26:19.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:19 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:19.876+0000 7fe04346f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T12:26:19.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:19 smithi016 ceph-mon[34212]: Deploying cephadm binary to smithi016 2024-09-17T12:26:19.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:19 smithi016 ceph-mon[34212]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:19.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:19 smithi016 ceph-mon[34212]: [17/Sep/2024:12:26:18] ENGINE Bus STARTING 2024-09-17T12:26:20.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:19 smithi196 ceph-mon[27428]: Deploying cephadm binary to smithi016 2024-09-17T12:26:20.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:19 smithi196 ceph-mon[27428]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:20.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:19 smithi196 ceph-mon[27428]: [17/Sep/2024:12:26:18] ENGINE Bus STARTING 2024-09-17T12:26:20.361 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:20.042+0000 7fe04346f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T12:26:20.361 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:20.121+0000 7fe04346f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T12:26:20.361 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:20.287+0000 7fe04346f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T12:26:20.361 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:20.358+0000 7fe04346f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T12:26:20.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:20.422+0000 7fe04346f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T12:26:20.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:20.594+0000 7fe04346f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T12:26:20.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:20.663+0000 7fe04346f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T12:26:20.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[23719]: [17/Sep/2024:12:26:18] ENGINE Serving on https://172.21.15.196:7150 2024-09-17T12:26:20.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[23719]: [17/Sep/2024:12:26:18] ENGINE Error in HTTPServer.serve 2024-09-17T12:26:20.941 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:26:20.941 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:26:20.941 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:26:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:26:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-09-17T12:26:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:26:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:26:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:26:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:26:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:26:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-09-17T12:26:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:26:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-09-17T12:26:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:26:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-09-17T12:26:20.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:26:20.943 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-09-17T12:26:20.943 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:26:20.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[23719]: Deploying cephadm binary to smithi196 2024-09-17T12:26:20.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[23719]: [17/Sep/2024:12:26:19] ENGINE Serving on http://172.21.15.196:8765 2024-09-17T12:26:20.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[23719]: [17/Sep/2024:12:26:19] ENGINE Bus STARTED 2024-09-17T12:26:20.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[23719]: mgrmap e26: x(active, since 3s) 2024-09-17T12:26:20.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:20.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:20.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[34212]: [17/Sep/2024:12:26:18] ENGINE Serving on https://172.21.15.196:7150 2024-09-17T12:26:20.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[34212]: [17/Sep/2024:12:26:18] ENGINE Error in HTTPServer.serve 2024-09-17T12:26:20.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:26:20.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:26:20.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:26:20.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:26:20.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: self._run(expiration_interval) 2024-09-17T12:26:20.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:26:20.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:26:20.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:26:20.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:26:20.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:26:20.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: s = self.context.wrap_socket( 2024-09-17T12:26:20.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:26:20.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: return self.sslsocket_class._create( 2024-09-17T12:26:20.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:26:20.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: self.do_handshake() 2024-09-17T12:26:20.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:26:20.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: self._sslobj.do_handshake() 2024-09-17T12:26:20.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:26:20.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[34212]: Deploying cephadm binary to smithi196 2024-09-17T12:26:20.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[34212]: [17/Sep/2024:12:26:19] ENGINE Serving on http://172.21.15.196:8765 2024-09-17T12:26:20.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[34212]: [17/Sep/2024:12:26:19] ENGINE Bus STARTED 2024-09-17T12:26:20.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[34212]: mgrmap e26: x(active, since 3s) 2024-09-17T12:26:20.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:20.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:20.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:20.734+0000 7fe04346f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T12:26:20.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:20.820+0000 7fe04346f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T12:26:20.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:20.882+0000 7fe04346f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T12:26:20.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:20 smithi196 ceph-mon[27428]: [17/Sep/2024:12:26:18] ENGINE Serving on https://172.21.15.196:7150 2024-09-17T12:26:20.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:20 smithi196 ceph-mon[27428]: [17/Sep/2024:12:26:18] ENGINE Error in HTTPServer.serve 2024-09-17T12:26:20.955 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-17T12:26:20.955 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._run(expiration_interval) 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: s = self.context.wrap_socket( 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.sslsocket_class._create( 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.do_handshake() 2024-09-17T12:26:20.956 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:26:20.957 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._sslobj.do_handshake() 2024-09-17T12:26:20.957 INFO:journalctl@ceph.mon.b.smithi196.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:26:20.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:20 smithi196 ceph-mon[27428]: Deploying cephadm binary to smithi196 2024-09-17T12:26:20.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:20 smithi196 ceph-mon[27428]: [17/Sep/2024:12:26:19] ENGINE Serving on http://172.21.15.196:8765 2024-09-17T12:26:20.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:20 smithi196 ceph-mon[27428]: [17/Sep/2024:12:26:19] ENGINE Bus STARTED 2024-09-17T12:26:20.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:20 smithi196 ceph-mon[27428]: mgrmap e26: x(active, since 3s) 2024-09-17T12:26:20.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:20 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:20.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:20 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:21.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:20 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:20.946+0000 7fe04346f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T12:26:21.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:21 smithi016 ceph-mon[23719]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:21.830 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:21 smithi016 ceph-mon[34212]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:21.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:21 smithi196 ceph-mon[27428]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:22.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:21 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:21.828+0000 7fe04346f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T12:26:22.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:21 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:26:21] ENGINE Bus STARTING 2024-09-17T12:26:22.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:21 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: CherryPy Checker: 2024-09-17T12:26:22.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:21 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: The Application mounted at '' has an empty config. 2024-09-17T12:26:22.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:21 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: 2024-09-17T12:26:22.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:21 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:26:21] ENGINE Serving on http://:::9283 2024-09-17T12:26:22.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:21 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: [17/Sep/2024:12:26:21] ENGINE Bus STARTED 2024-09-17T12:26:22.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:22 smithi196 ceph-mon[27428]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:26:22.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:22 smithi196 ceph-mon[27428]: mgrmap e27: x(active, since 5s) 2024-09-17T12:26:22.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:22 smithi196 ceph-mon[27428]: Standby manager daemon y started 2024-09-17T12:26:22.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:22 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.16:0/3466571930' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T12:26:22.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:22 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.16:0/3466571930' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:26:22.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:22 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.16:0/3466571930' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T12:26:22.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:22 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.16:0/3466571930' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:26:22.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[23719]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:26:22.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[23719]: mgrmap e27: x(active, since 5s) 2024-09-17T12:26:22.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[23719]: Standby manager daemon y started 2024-09-17T12:26:22.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.16:0/3466571930' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T12:26:22.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.16:0/3466571930' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:26:22.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.16:0/3466571930' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T12:26:22.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.16:0/3466571930' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:26:22.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[34212]: from='client.24694 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:26:22.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[34212]: mgrmap e27: x(active, since 5s) 2024-09-17T12:26:22.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[34212]: Standby manager daemon y started 2024-09-17T12:26:22.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.16:0/3466571930' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T12:26:22.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.16:0/3466571930' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:26:22.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.16:0/3466571930' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T12:26:22.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:22 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.16:0/3466571930' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:26:23.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:23 smithi016 ceph-mon[34212]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-17T12:26:23.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:23 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:26:23.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:23 smithi016 ceph-mon[34212]: mgrmap e28: x(active, since 6s), standbys: y 2024-09-17T12:26:23.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:23 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:23.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:23 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:23.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:23 smithi016 ceph-mon[23719]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-17T12:26:23.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:23 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:26:23.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:23 smithi016 ceph-mon[23719]: mgrmap e28: x(active, since 6s), standbys: y 2024-09-17T12:26:23.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:23 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:23.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:23 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:23 smithi196 ceph-mon[27428]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-17T12:26:24.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:23 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:26:24.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:23 smithi196 ceph-mon[27428]: mgrmap e28: x(active, since 6s), standbys: y 2024-09-17T12:26:24.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:23 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:23 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.504 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:26:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:24 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:24.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:24.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:24 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:25.087 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:26:25.088 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (4m) 1s ago 5m 26.0M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:26:25.088 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (5m) 1s ago 5m 88.6M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:26:25.088 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (3m) 1s ago 3m 57.4M - 3.8 980a0bfcd684 bc9ab4f22b6b 2024-09-17T12:26:25.088 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (27s) 1s ago 9m 497M - 19.3.0-4937-g74984932 6c1fcab85602 7ba30e0f95c9 2024-09-17T12:26:25.088 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:9283,8765 running (10m) 1s ago 10m 302M - 18.2.4-855-g19e2a2be 980a0bfcd684 caffea02f791 2024-09-17T12:26:25.088 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (10m) 1s ago 10m 56.2M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 5d21f110d6fc 2024-09-17T12:26:25.088 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (9m) 1s ago 9m 44.2M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 262ef622777e 2024-09-17T12:26:25.088 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (9m) 1s ago 9m 49.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 950b72634364 2024-09-17T12:26:25.088 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (4m) 1s ago 4m 15.9M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:26:25.088 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (4m) 1s ago 4m 16.0M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:26:25.088 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (8m) 1s ago 8m 61.2M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 e30aaffd3b5f 2024-09-17T12:26:25.088 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (8m) 1s ago 8m 53.5M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 3ad94c30c9c9 2024-09-17T12:26:25.089 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (8m) 1s ago 8m 54.3M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 1fa310695876 2024-09-17T12:26:25.089 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (7m) 1s ago 7m 58.9M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 4a4f8eeaff13 2024-09-17T12:26:25.089 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (7m) 1s ago 7m 54.1M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 afa20056b398 2024-09-17T12:26:25.089 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (6m) 1s ago 6m 50.0M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 f0dcabb8902f 2024-09-17T12:26:25.089 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (6m) 1s ago 6m 57.8M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:26:25.089 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (6m) 1s ago 6m 60.6M 4016M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:26:25.089 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (4m) 1s ago 5m 52.3M - 2.43.0 a07b618ecd1d cb4f31b0d350 2024-09-17T12:26:25.089 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (3m) 1s ago 3m 77.7M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:26:25.089 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (4m) 1s ago 4m 78.4M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:26:25.089 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (3m) 1s ago 3m 79.6M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:26:25.089 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (3m) 1s ago 3m 77.4M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:26:25.443 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:26:25.443 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:26:25.443 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T12:26:25.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:26:25.443 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:26:25.443 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1, 2024-09-17T12:26:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 1 2024-09-17T12:26:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:26:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:26:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T12:26:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:26:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:26:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:26:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:26:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:26:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 16, 2024-09-17T12:26:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 1 2024-09-17T12:26:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:26:25.444 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:26:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:26:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:26:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:26:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:26:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-09-17T12:26:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "1/23 daemons upgraded", 2024-09-17T12:26:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "", 2024-09-17T12:26:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:26:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:26:25.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:25 smithi196 ceph-mon[27428]: Adjusting osd_memory_target on smithi016 to 2737M 2024-09-17T12:26:25.967 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:25 smithi196 ceph-mon[27428]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:25.967 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:25 smithi196 ceph-mon[27428]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T12:26:25.967 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:25 smithi196 ceph-mon[27428]: from='client.25082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:25.967 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:25 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3917422919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:26:26.088 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-09-17T12:26:26.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:25 smithi016 ceph-mon[23719]: Adjusting osd_memory_target on smithi016 to 2737M 2024-09-17T12:26:26.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:25 smithi016 ceph-mon[23719]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:26.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:25 smithi016 ceph-mon[23719]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T12:26:26.089 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:25 smithi016 ceph-mon[23719]: from='client.25082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:26.089 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:25 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3917422919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:26:26.089 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:25 smithi016 ceph-mon[34212]: Adjusting osd_memory_target on smithi016 to 2737M 2024-09-17T12:26:26.089 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:25 smithi016 ceph-mon[34212]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:26.089 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:25 smithi016 ceph-mon[34212]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T12:26:26.089 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:25 smithi016 ceph-mon[34212]: from='client.25082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:26.089 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:25 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3917422919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:26:26.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[23946]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:26:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-17T12:26:26.974 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:26 smithi196 ceph-mon[27428]: from='client.25088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:26.974 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:26 smithi196 ceph-mon[27428]: from='client.25100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:26.975 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:26 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/192614856' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:26:27.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:26 smithi016 ceph-mon[23719]: from='client.25088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:27.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:26 smithi016 ceph-mon[23719]: from='client.25100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:27.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:26 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/192614856' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:26:27.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:26 smithi016 ceph-mon[34212]: from='client.25088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:27.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:26 smithi016 ceph-mon[34212]: from='client.25100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:27.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:26 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/192614856' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:26:28.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-17T12:26:28.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:28.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:27 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:26:28.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-17T12:26:28.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:28.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:26:28.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-17T12:26:28.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T12:26:28.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:28.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:27 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:26:28.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: Adjusting osd_memory_target on smithi196 to 2992M 2024-09-17T12:26:28.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:26:28.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:26:28.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:26:28.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:26:28.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:26:28.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:26:28.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:26:28.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:26:28.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:26:28.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:26:28.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:28.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: Adjusting osd_memory_target on smithi196 to 2992M 2024-09-17T12:26:28.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:26:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:26:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:26:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:26:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:26:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:26:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:26:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:26:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:26:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:26:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:28 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:29.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: Adjusting osd_memory_target on smithi196 to 2992M 2024-09-17T12:26:29.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:26:29.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:26:29.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:26:29.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:26:29.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:26:29.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:26:29.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:26:29.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:26:29.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:29.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:29.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:29.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:29.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:29.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:26:29.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:26:29.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:28 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:30.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:29 smithi196 ceph-mon[27428]: Reconfiguring iscsi.foo.smithi016.qaboca (dependencies changed)... 2024-09-17T12:26:30.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:29 smithi196 ceph-mon[27428]: Reconfiguring daemon iscsi.foo.smithi016.qaboca on smithi016 2024-09-17T12:26:30.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:29 smithi196 ceph-mon[27428]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-17T12:26:30.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:29 smithi016 ceph-mon[23719]: Reconfiguring iscsi.foo.smithi016.qaboca (dependencies changed)... 2024-09-17T12:26:30.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:29 smithi016 ceph-mon[23719]: Reconfiguring daemon iscsi.foo.smithi016.qaboca on smithi016 2024-09-17T12:26:30.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:29 smithi016 ceph-mon[23719]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-17T12:26:30.057 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:29 smithi016 ceph-mon[34212]: Reconfiguring iscsi.foo.smithi016.qaboca (dependencies changed)... 2024-09-17T12:26:30.057 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:29 smithi016 ceph-mon[34212]: Reconfiguring daemon iscsi.foo.smithi016.qaboca on smithi016 2024-09-17T12:26:30.057 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:29 smithi016 ceph-mon[34212]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-17T12:26:31.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:31 smithi016 ceph-mon[23719]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:26:31.785 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:31 smithi016 ceph-mon[34212]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:26:32.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-mon[27428]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:26:32.026 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: 2024-09-17T12:26:31.814+0000 7fa97d3d3640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.qaboca ... 2024-09-17T12:26:32.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Creating ceph-iscsi config... 2024-09-17T12:26:32.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/iscsi-gateway.cfg 2024-09-17T12:26:32.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/tcmu-runner-entrypoint.sh 2024-09-17T12:26:32.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Non-zero exit code 1 from systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca 2024-09-17T12:26:32.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: systemctl: stderr Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:32.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: systemctl: stderr See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:32.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Traceback (most recent call last): 2024-09-17T12:26:32.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T12:26:32.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: return _run_code(code, main_globals, None, 2024-09-17T12:26:32.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T12:26:32.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: exec(code, run_globals) 2024-09-17T12:26:32.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5628, in 2024-09-17T12:26:32.028 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5616, in main 2024-09-17T12:26:32.028 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-17T12:26:32.028 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-17T12:26:32.028 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-17T12:26:32.028 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-17T12:26:32.028 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T12:26:32.028 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: RuntimeError: Failed command: systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca: Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:32.028 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:32.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Traceback (most recent call last): 2024-09-17T12:26:32.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-17T12:26:32.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T12:26:32.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2557, in _daemon_action 2024-09-17T12:26:32.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: return self.wait_async( 2024-09-17T12:26:32.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-17T12:26:32.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: return self.event_loop.get_result(coro, timeout) 2024-09-17T12:26:32.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T12:26:32.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: return future.result(timeout) 2024-09-17T12:26:32.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T12:26:32.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: return self.__get_result() 2024-09-17T12:26:32.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T12:26:32.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: raise self._exception 2024-09-17T12:26:32.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-17T12:26:32.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: out, err, code = await self._run_cephadm( 2024-09-17T12:26:32.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-17T12:26:32.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: raise OrchestratorError( 2024-09-17T12:26:32.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.qaboca ... 2024-09-17T12:26:32.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Creating ceph-iscsi config... 2024-09-17T12:26:32.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/iscsi-gateway.cfg 2024-09-17T12:26:32.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/tcmu-runner-entrypoint.sh 2024-09-17T12:26:32.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Non-zero exit code 1 from systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca 2024-09-17T12:26:32.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: systemctl: stderr Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:32.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: systemctl: stderr See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:32.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Traceback (most recent call last): 2024-09-17T12:26:32.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T12:26:32.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: return _run_code(code, main_globals, None, 2024-09-17T12:26:32.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T12:26:32.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: exec(code, run_globals) 2024-09-17T12:26:32.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5628, in 2024-09-17T12:26:32.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5616, in main 2024-09-17T12:26:32.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-17T12:26:32.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-17T12:26:32.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-17T12:26:32.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-17T12:26:32.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T12:26:32.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: RuntimeError: Failed command: systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca: Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:32.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:33.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:32 smithi196 ceph-mon[27428]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.qaboca ... 2024-09-17T12:26:33.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-09-17T12:26:33.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/iscsi-gateway.cfg 2024-09-17T12:26:33.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/tcmu-runner-entrypoint.sh 2024-09-17T12:26:33.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca 2024-09-17T12:26:33.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:33.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:33.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-17T12:26:33.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T12:26:33.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-09-17T12:26:33.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T12:26:33.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-09-17T12:26:33.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5628, in 2024-09-17T12:26:33.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5616, in main 2024-09-17T12:26:33.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-17T12:26:33.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-17T12:26:33.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-17T12:26:33.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-17T12:26:33.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T12:26:33.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca: Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:33.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:33.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-17T12:26:33.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-17T12:26:33.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T12:26:33.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2557, in _daemon_action 2024-09-17T12:26:33.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.wait_async( 2024-09-17T12:26:33.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-17T12:26:33.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T12:26:33.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T12:26:33.029 INFO:journalctl@ceph.mon.b.smithi196.stdout: return future.result(timeout) 2024-09-17T12:26:33.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T12:26:33.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.__get_result() 2024-09-17T12:26:33.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T12:26:33.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: raise self._exception 2024-09-17T12:26:33.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-17T12:26:33.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: out, err, code = await self._run_cephadm( 2024-09-17T12:26:33.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-17T12:26:33.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: raise OrchestratorError( 2024-09-17T12:26:33.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.qaboca ... 2024-09-17T12:26:33.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-09-17T12:26:33.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/iscsi-gateway.cfg 2024-09-17T12:26:33.030 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/tcmu-runner-entrypoint.sh 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5628, in 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5616, in main 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-17T12:26:33.031 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-17T12:26:33.032 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T12:26:33.032 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca: Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:33.032 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:33.032 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:32 smithi196 ceph-mon[27428]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T12:26:33.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:32 smithi016 ceph-mon[34212]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.qaboca ... 2024-09-17T12:26:33.191 INFO:journalctl@ceph.mon.c.smithi016.stdout: Creating ceph-iscsi config... 2024-09-17T12:26:33.192 INFO:journalctl@ceph.mon.c.smithi016.stdout: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/iscsi-gateway.cfg 2024-09-17T12:26:33.192 INFO:journalctl@ceph.mon.c.smithi016.stdout: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/tcmu-runner-entrypoint.sh 2024-09-17T12:26:33.192 INFO:journalctl@ceph.mon.c.smithi016.stdout: Non-zero exit code 1 from systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca 2024-09-17T12:26:33.192 INFO:journalctl@ceph.mon.c.smithi016.stdout: systemctl: stderr Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:33.192 INFO:journalctl@ceph.mon.c.smithi016.stdout: systemctl: stderr See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:33.192 INFO:journalctl@ceph.mon.c.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:26:33.192 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T12:26:33.192 INFO:journalctl@ceph.mon.c.smithi016.stdout: return _run_code(code, main_globals, None, 2024-09-17T12:26:33.192 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T12:26:33.192 INFO:journalctl@ceph.mon.c.smithi016.stdout: exec(code, run_globals) 2024-09-17T12:26:33.192 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5628, in 2024-09-17T12:26:33.193 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5616, in main 2024-09-17T12:26:33.193 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-17T12:26:33.193 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-17T12:26:33.193 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-17T12:26:33.193 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-17T12:26:33.193 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T12:26:33.193 INFO:journalctl@ceph.mon.c.smithi016.stdout: RuntimeError: Failed command: systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca: Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:33.193 INFO:journalctl@ceph.mon.c.smithi016.stdout: See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:33.193 INFO:journalctl@ceph.mon.c.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:26:33.193 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-17T12:26:33.193 INFO:journalctl@ceph.mon.c.smithi016.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T12:26:33.193 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2557, in _daemon_action 2024-09-17T12:26:33.193 INFO:journalctl@ceph.mon.c.smithi016.stdout: return self.wait_async( 2024-09-17T12:26:33.194 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-17T12:26:33.194 INFO:journalctl@ceph.mon.c.smithi016.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T12:26:33.194 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T12:26:33.194 INFO:journalctl@ceph.mon.c.smithi016.stdout: return future.result(timeout) 2024-09-17T12:26:33.194 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T12:26:33.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:32 smithi016 ceph-mon[23719]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.qaboca ... 2024-09-17T12:26:33.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: Creating ceph-iscsi config... 2024-09-17T12:26:33.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/iscsi-gateway.cfg 2024-09-17T12:26:33.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/tcmu-runner-entrypoint.sh 2024-09-17T12:26:33.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: Non-zero exit code 1 from systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca 2024-09-17T12:26:33.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: systemctl: stderr Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:33.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: systemctl: stderr See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:33.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:26:33.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T12:26:33.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: return _run_code(code, main_globals, None, 2024-09-17T12:26:33.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T12:26:33.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: exec(code, run_globals) 2024-09-17T12:26:33.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5628, in 2024-09-17T12:26:33.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5616, in main 2024-09-17T12:26:33.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-17T12:26:33.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-17T12:26:33.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-17T12:26:33.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-17T12:26:33.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T12:26:33.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: RuntimeError: Failed command: systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca: Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:33.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:33.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:26:33.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-17T12:26:33.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T12:26:33.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2557, in _daemon_action 2024-09-17T12:26:33.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.wait_async( 2024-09-17T12:26:33.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-17T12:26:33.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T12:26:33.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T12:26:33.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: return future.result(timeout) 2024-09-17T12:26:33.197 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T12:26:33.197 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.__get_result() 2024-09-17T12:26:33.197 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T12:26:33.197 INFO:journalctl@ceph.mon.a.smithi016.stdout: raise self._exception 2024-09-17T12:26:33.197 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-17T12:26:33.197 INFO:journalctl@ceph.mon.a.smithi016.stdout: out, err, code = await self._run_cephadm( 2024-09-17T12:26:33.197 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-17T12:26:33.197 INFO:journalctl@ceph.mon.a.smithi016.stdout: raise OrchestratorError( 2024-09-17T12:26:33.197 INFO:journalctl@ceph.mon.a.smithi016.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.qaboca ... 2024-09-17T12:26:33.197 INFO:journalctl@ceph.mon.a.smithi016.stdout: Creating ceph-iscsi config... 2024-09-17T12:26:33.197 INFO:journalctl@ceph.mon.a.smithi016.stdout: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/iscsi-gateway.cfg 2024-09-17T12:26:33.197 INFO:journalctl@ceph.mon.a.smithi016.stdout: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/tcmu-runner-entrypoint.sh 2024-09-17T12:26:33.197 INFO:journalctl@ceph.mon.a.smithi016.stdout: Non-zero exit code 1 from systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca 2024-09-17T12:26:33.198 INFO:journalctl@ceph.mon.a.smithi016.stdout: systemctl: stderr Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:33.198 INFO:journalctl@ceph.mon.a.smithi016.stdout: systemctl: stderr See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:33.198 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:26:33.198 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T12:26:33.198 INFO:journalctl@ceph.mon.a.smithi016.stdout: return _run_code(code, main_globals, None, 2024-09-17T12:26:33.198 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T12:26:33.198 INFO:journalctl@ceph.mon.a.smithi016.stdout: exec(code, run_globals) 2024-09-17T12:26:33.198 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5628, in 2024-09-17T12:26:33.198 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5616, in main 2024-09-17T12:26:33.199 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-17T12:26:33.199 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-17T12:26:33.199 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-17T12:26:33.199 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-17T12:26:33.199 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T12:26:33.199 INFO:journalctl@ceph.mon.a.smithi016.stdout: RuntimeError: Failed command: systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca: Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:33.199 INFO:journalctl@ceph.mon.a.smithi016.stdout: See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:33.199 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:32 smithi016 ceph-mon[23719]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T12:26:33.199 INFO:journalctl@ceph.mon.c.smithi016.stdout: return self.__get_result() 2024-09-17T12:26:33.199 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T12:26:33.199 INFO:journalctl@ceph.mon.c.smithi016.stdout: raise self._exception 2024-09-17T12:26:33.200 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-17T12:26:33.200 INFO:journalctl@ceph.mon.c.smithi016.stdout: out, err, code = await self._run_cephadm( 2024-09-17T12:26:33.200 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-17T12:26:33.200 INFO:journalctl@ceph.mon.c.smithi016.stdout: raise OrchestratorError( 2024-09-17T12:26:33.200 INFO:journalctl@ceph.mon.c.smithi016.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.qaboca ... 2024-09-17T12:26:33.200 INFO:journalctl@ceph.mon.c.smithi016.stdout: Creating ceph-iscsi config... 2024-09-17T12:26:33.200 INFO:journalctl@ceph.mon.c.smithi016.stdout: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/iscsi-gateway.cfg 2024-09-17T12:26:33.200 INFO:journalctl@ceph.mon.c.smithi016.stdout: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/tcmu-runner-entrypoint.sh 2024-09-17T12:26:33.200 INFO:journalctl@ceph.mon.c.smithi016.stdout: Non-zero exit code 1 from systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca 2024-09-17T12:26:33.200 INFO:journalctl@ceph.mon.c.smithi016.stdout: systemctl: stderr Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:33.200 INFO:journalctl@ceph.mon.c.smithi016.stdout: systemctl: stderr See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:33.200 INFO:journalctl@ceph.mon.c.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:26:33.201 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T12:26:33.201 INFO:journalctl@ceph.mon.c.smithi016.stdout: return _run_code(code, main_globals, None, 2024-09-17T12:26:33.201 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T12:26:33.201 INFO:journalctl@ceph.mon.c.smithi016.stdout: exec(code, run_globals) 2024-09-17T12:26:33.201 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5628, in 2024-09-17T12:26:33.201 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5616, in main 2024-09-17T12:26:33.201 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-17T12:26:33.201 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-17T12:26:33.201 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-17T12:26:33.201 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-17T12:26:33.201 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T12:26:33.201 INFO:journalctl@ceph.mon.c.smithi016.stdout: RuntimeError: Failed command: systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca: Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:33.202 INFO:journalctl@ceph.mon.c.smithi016.stdout: See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:33.202 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:32 smithi016 ceph-mon[34212]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T12:26:33.747 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:33 smithi196 ceph-mon[27428]: Reconfiguring daemon prometheus.a on smithi196 2024-09-17T12:26:34.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:33 smithi196 ceph-mon[27428]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:26:34.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:33 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:34.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:33 smithi016 ceph-mon[23719]: Reconfiguring daemon prometheus.a on smithi196 2024-09-17T12:26:34.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:33 smithi016 ceph-mon[23719]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:26:34.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:33 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:34.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:33 smithi016 ceph-mon[34212]: Reconfiguring daemon prometheus.a on smithi196 2024-09-17T12:26:34.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:33 smithi016 ceph-mon[34212]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:26:34.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:33 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:35.247 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:34 smithi196 systemd[1]: Stopping Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:26:35.527 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:26:35.255Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T12:26:35.527 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:26:35.255Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-17T12:26:35.527 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:26:35.255Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-17T12:26:35.527 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:26:35.255Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T12:26:35.527 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:26:35.255Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-17T12:26:35.527 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:26:35.255Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-17T12:26:35.527 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:26:35.255Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T12:26:35.527 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:26:35.255Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-17T12:26:35.527 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:26:35.255Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-17T12:26:35.527 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:26:35.256Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T12:26:35.528 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:26:35.256Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-17T12:26:35.528 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[57536]: ts=2024-09-17T12:26:35.256Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-17T12:26:35.528 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 podman[70573]: 2024-09-17 12:26:35.265838722 +0000 UTC m=+0.119857388 container died cb4f31b0d3509a34e9393e660fdd591b69131463e39da2fcf5a357f0ffadbb5e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:26:35.933 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:35 smithi196 ceph-mon[27428]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:26:35.933 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 podman[70573]: 2024-09-17 12:26:35.646156918 +0000 UTC m=+0.500175586 container cleanup cb4f31b0d3509a34e9393e660fdd591b69131463e39da2fcf5a357f0ffadbb5e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:26:35.933 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 bash[70573]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T12:26:36.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:35 smithi016 ceph-mon[23719]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:26:36.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:35 smithi016 ceph-mon[34212]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:26:36.276 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:35 smithi196 podman[70588]: 2024-09-17 12:26:35.930862708 +0000 UTC m=+0.665234119 container remove cb4f31b0d3509a34e9393e660fdd591b69131463e39da2fcf5a357f0ffadbb5e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:26:36.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T12:26:36.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 systemd[1]: Stopped Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:26:36.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.928s CPU time. 2024-09-17T12:26:36.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 systemd[1]: Starting Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:26:36.560 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 podman[70680]: 2024-09-17 12:26:36.180904277 +0000 UTC m=+0.016710524 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-17T12:26:36.560 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 podman[70680]: 2024-09-17 12:26:36.315266274 +0000 UTC m=+0.151072513 container create 65e20131150468bd6e7308983c45ea00fa892076b471c4c7b1c009f004ff8ce2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:26:36.560 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 podman[70680]: 2024-09-17 12:26:36.559272012 +0000 UTC m=+0.395078250 container init 65e20131150468bd6e7308983c45ea00fa892076b471c4c7b1c009f004ff8ce2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:26:37.027 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 podman[70680]: 2024-09-17 12:26:36.562279664 +0000 UTC m=+0.398085909 container start 65e20131150468bd6e7308983c45ea00fa892076b471c4c7b1c009f004ff8ce2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:26:37.027 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.585Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-17T12:26:37.027 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.586Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-17T12:26:37.027 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.586Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi196 (none))" 2024-09-17T12:26:37.027 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.586Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T12:26:37.027 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.586Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T12:26:37.027 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.587Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-17T12:26:37.028 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.588Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-17T12:26:37.028 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.589Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-17T12:26:37.028 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.589Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-17T12:26:37.028 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.590Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-17T12:26:37.028 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.590Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.432µs 2024-09-17T12:26:37.028 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.590Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-17T12:26:37.028 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.593Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-17T12:26:37.028 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.600Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-17T12:26:37.028 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.601Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-17T12:26:37.028 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.601Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.158µs wal_replay_duration=10.477079ms wbl_replay_duration=118ns total_replay_duration=10.513346ms 2024-09-17T12:26:37.028 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.603Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-17T12:26:37.028 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.603Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-17T12:26:37.029 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.603Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T12:26:37.029 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.622Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.004768ms db_storage=862ns remote_storage=1.154µs web_handler=431ns query_engine=685ns scrape=452.926µs scrape_sd=131.31µs notify=17.659µs notify_sd=17.875µs rules=17.991325ms tracing=5.341µs 2024-09-17T12:26:37.029 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.622Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-17T12:26:37.029 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:26:36.622Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-17T12:26:37.029 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 bash[70680]: 65e20131150468bd6e7308983c45ea00fa892076b471c4c7b1c009f004ff8ce2 2024-09-17T12:26:37.029 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:26:36 smithi196 systemd[1]: Started Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:26:38.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:37 smithi196 ceph-mon[27428]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T12:26:38.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:37 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:38.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:37 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:38.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:37 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:26:38.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:37 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:26:38.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:37 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:38.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:37 smithi196 ceph-mon[27428]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T12:26:38.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:37 smithi196 ceph-mon[27428]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T12:26:38.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[23719]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T12:26:38.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:38.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:38.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:26:38.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[23719]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:26:38.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:38.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[23719]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T12:26:38.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[23719]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T12:26:38.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[34212]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T12:26:38.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:38.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:38.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:26:38.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:26:38.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:38.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[34212]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T12:26:38.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:37 smithi016 ceph-mon[34212]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T12:26:39.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[23719]: Upgrade: Updating mgr.y 2024-09-17T12:26:39.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:39.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:26:39.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:26:39.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:26:39.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:39.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[23719]: Deploying daemon mgr.y on smithi016 2024-09-17T12:26:39.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[23719]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T12:26:39.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[34212]: Upgrade: Updating mgr.y 2024-09-17T12:26:39.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:39.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:26:39.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:26:39.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:26:39.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:39.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[34212]: Deploying daemon mgr.y on smithi016 2024-09-17T12:26:39.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:39 smithi016 ceph-mon[34212]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T12:26:39.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:39 smithi196 ceph-mon[27428]: Upgrade: Updating mgr.y 2024-09-17T12:26:39.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:39 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:39.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:39 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:26:39.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:39 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:26:39.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:39 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:26:39.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:39 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:39.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:39 smithi196 ceph-mon[27428]: Deploying daemon mgr.y on smithi016 2024-09-17T12:26:39.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:39 smithi196 ceph-mon[27428]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T12:26:40.108 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:39 smithi016 systemd[1]: Stopping Ceph mgr.y for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:26:40.402 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:40 smithi016 podman[78938]: 2024-09-17 12:26:40.106481481 +0000 UTC m=+0.129989428 container died caffea02f7912ca615199d832616180397687a9a0ef702d231d4eea5f7ef2078 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux ) 2024-09-17T12:26:40.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:40 smithi016 podman[78938]: 2024-09-17 12:26:40.475546567 +0000 UTC m=+0.499054511 container cleanup caffea02f7912ca615199d832616180397687a9a0ef702d231d4eea5f7ef2078 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-17T12:26:40.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:40 smithi016 bash[78938]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y 2024-09-17T12:26:41.104 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:40 smithi016 podman[78950]: 2024-09-17 12:26:40.744245504 +0000 UTC m=+0.639687788 container remove caffea02f7912ca615199d832616180397687a9a0ef702d231d4eea5f7ef2078 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-17T12:26:41.104 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:40 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-17T12:26:41.104 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:40 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-09-17T12:26:41.104 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:40 smithi016 systemd[1]: Stopped Ceph mgr.y for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:26:41.105 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:40 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.y.service: Consumed 49.722s CPU time. 2024-09-17T12:26:41.363 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:41 smithi016 systemd[1]: Starting Ceph mgr.y for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:26:41.363 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:41 smithi016 podman[79078]: 2024-09-17 12:26:41.219551221 +0000 UTC m=+0.016517027 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:26:41.363 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:41 smithi016 podman[79078]: 2024-09-17 12:26:41.362146716 +0000 UTC m=+0.159112541 container create 953d9c4b1666ffecc5bd796e5149e6bfb8e239107f773f99dfe564c29c44a8e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-7498493, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-17T12:26:41.627 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:41 smithi016 podman[79078]: 2024-09-17 12:26:41.598255317 +0000 UTC m=+0.395221120 container init 953d9c4b1666ffecc5bd796e5149e6bfb8e239107f773f99dfe564c29c44a8e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-7498493) 2024-09-17T12:26:41.627 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:41 smithi016 podman[79078]: 2024-09-17 12:26:41.601184104 +0000 UTC m=+0.398149896 container start 953d9c4b1666ffecc5bd796e5149e6bfb8e239107f773f99dfe564c29c44a8e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-7498493, maintainer=Guillaume Abrioux ) 2024-09-17T12:26:41.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:41 smithi016 ceph-mon[34212]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T12:26:41.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:41 smithi016 ceph-mon[23719]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T12:26:41.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:41 smithi016 ceph-mgr[79096]: -- 172.21.15.16:0/686976232 <== mon.2 v2:172.21.15.16:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564c7bf8b0e0 con 0x564c7bf69400 2024-09-17T12:26:41.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:41 smithi016 bash[79078]: 953d9c4b1666ffecc5bd796e5149e6bfb8e239107f773f99dfe564c29c44a8e8 2024-09-17T12:26:41.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:41 smithi016 systemd[1]: Started Ceph mgr.y for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:26:42.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:41 smithi196 ceph-mon[27428]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T12:26:42.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:42 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:42.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:42 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:42.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:42 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:42.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:42 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:42.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:42 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:42.760 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:42 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:42.760 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:42 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:42.760 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:42 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:42.760 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:42 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:42.760 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:42 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:43.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:42 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:43.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:42 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:43.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:42 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:43.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:42 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:43.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:42 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:43.977 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:43 smithi016 ceph-mon[23719]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T12:26:43.978 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:43 smithi016 ceph-mon[34212]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T12:26:44.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:43 smithi196 ceph-mon[27428]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T12:26:45.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:44 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3881813015' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T12:26:45.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:44 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/4030615206' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/669007717"}]: dispatch 2024-09-17T12:26:45.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:44 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3881813015' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T12:26:45.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:44 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/4030615206' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/669007717"}]: dispatch 2024-09-17T12:26:45.064 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:44 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3881813015' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T12:26:45.064 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:44 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/4030615206' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/669007717"}]: dispatch 2024-09-17T12:26:46.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:45 smithi196 ceph-mon[27428]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T12:26:46.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:45 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/4030615206' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/669007717"}]': finished 2024-09-17T12:26:46.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:45 smithi196 ceph-mon[27428]: osdmap e96: 8 total, 8 up, 8 in 2024-09-17T12:26:46.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:45 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/941042066' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1859821668"}]: dispatch 2024-09-17T12:26:46.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:45 smithi016 ceph-mon[23719]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T12:26:46.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:45 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/4030615206' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/669007717"}]': finished 2024-09-17T12:26:46.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:45 smithi016 ceph-mon[23719]: osdmap e96: 8 total, 8 up, 8 in 2024-09-17T12:26:46.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:45 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/941042066' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1859821668"}]: dispatch 2024-09-17T12:26:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:45 smithi016 ceph-mon[34212]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T12:26:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:45 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/4030615206' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/669007717"}]': finished 2024-09-17T12:26:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:45 smithi016 ceph-mon[34212]: osdmap e96: 8 total, 8 up, 8 in 2024-09-17T12:26:46.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:45 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/941042066' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1859821668"}]: dispatch 2024-09-17T12:26:46.779 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:46 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/941042066' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1859821668"}]': finished 2024-09-17T12:26:46.779 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:46 smithi016 ceph-mon[23719]: osdmap e97: 8 total, 8 up, 8 in 2024-09-17T12:26:46.780 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:46 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1592401080' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3734045536"}]: dispatch 2024-09-17T12:26:46.780 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:46 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3734045536"}]: dispatch 2024-09-17T12:26:46.780 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:46 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:46.780 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:46 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:46.780 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:46 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/941042066' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1859821668"}]': finished 2024-09-17T12:26:46.780 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:46 smithi016 ceph-mon[34212]: osdmap e97: 8 total, 8 up, 8 in 2024-09-17T12:26:46.780 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:46 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1592401080' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3734045536"}]: dispatch 2024-09-17T12:26:46.780 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:46 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3734045536"}]: dispatch 2024-09-17T12:26:46.781 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:46 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:46.781 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:46 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:47.026 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:26:46] "GET /metrics HTTP/1.1" 200 38433 "" "Prometheus/2.43.0" 2024-09-17T12:26:47.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:46 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/941042066' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1859821668"}]': finished 2024-09-17T12:26:47.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:46 smithi196 ceph-mon[27428]: osdmap e97: 8 total, 8 up, 8 in 2024-09-17T12:26:47.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:46 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1592401080' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3734045536"}]: dispatch 2024-09-17T12:26:47.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:46 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3734045536"}]: dispatch 2024-09-17T12:26:47.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:46 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:47.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:46 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:47.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:26:47] ENGINE Bus STARTING 2024-09-17T12:26:47.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: CherryPy Checker: 2024-09-17T12:26:47.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: The Application mounted at '' has an empty config. 2024-09-17T12:26:47.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:26:47.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:26:47] ENGINE Serving on http://:::9283 2024-09-17T12:26:47.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:26:47] ENGINE Bus STARTED 2024-09-17T12:26:48.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3734045536"}]': finished 2024-09-17T12:26:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: osdmap e98: 8 total, 8 up, 8 in 2024-09-17T12:26:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3985861153' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2599403596"}]: dispatch 2024-09-17T12:26:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2599403596"}]: dispatch 2024-09-17T12:26:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: Standby manager daemon y restarted 2024-09-17T12:26:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: Standby manager daemon y started 2024-09-17T12:26:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T12:26:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:26:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T12:26:48.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:26:48.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:48.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:47 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:48.073 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:48.073 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3734045536"}]': finished 2024-09-17T12:26:48.073 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: osdmap e98: 8 total, 8 up, 8 in 2024-09-17T12:26:48.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:48.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:48.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3985861153' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2599403596"}]: dispatch 2024-09-17T12:26:48.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2599403596"}]: dispatch 2024-09-17T12:26:48.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: Standby manager daemon y restarted 2024-09-17T12:26:48.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: Standby manager daemon y started 2024-09-17T12:26:48.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T12:26:48.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:26:48.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T12:26:48.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:26:48.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:48.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:48.075 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:48.075 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3734045536"}]': finished 2024-09-17T12:26:48.075 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: osdmap e98: 8 total, 8 up, 8 in 2024-09-17T12:26:48.075 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:48.075 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:48.075 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3985861153' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2599403596"}]: dispatch 2024-09-17T12:26:48.075 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2599403596"}]: dispatch 2024-09-17T12:26:48.075 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: Standby manager daemon y restarted 2024-09-17T12:26:48.075 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: Standby manager daemon y started 2024-09-17T12:26:48.075 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T12:26:48.075 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:26:48.075 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T12:26:48.076 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:26:48.076 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:48.076 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:47 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:49.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:48 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:49.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:48 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2599403596"}]': finished 2024-09-17T12:26:49.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:48 smithi016 ceph-mon[23719]: osdmap e99: 8 total, 8 up, 8 in 2024-09-17T12:26:49.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:48 smithi016 ceph-mon[23719]: mgrmap e29: x(active, since 31s), standbys: y 2024-09-17T12:26:49.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:48 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2925162511' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2599403596"}]: dispatch 2024-09-17T12:26:49.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:48 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:49.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:48 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2599403596"}]': finished 2024-09-17T12:26:49.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:48 smithi196 ceph-mon[27428]: osdmap e99: 8 total, 8 up, 8 in 2024-09-17T12:26:49.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:48 smithi196 ceph-mon[27428]: mgrmap e29: x(active, since 31s), standbys: y 2024-09-17T12:26:49.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:48 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2925162511' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2599403596"}]: dispatch 2024-09-17T12:26:49.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:48 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:49.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:48 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2599403596"}]': finished 2024-09-17T12:26:49.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:48 smithi016 ceph-mon[34212]: osdmap e99: 8 total, 8 up, 8 in 2024-09-17T12:26:49.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:48 smithi016 ceph-mon[34212]: mgrmap e29: x(active, since 31s), standbys: y 2024-09-17T12:26:49.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:48 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2925162511' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2599403596"}]: dispatch 2024-09-17T12:26:50.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:49 smithi196 ceph-mon[27428]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:50.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:49 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2925162511' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2599403596"}]': finished 2024-09-17T12:26:50.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:49 smithi196 ceph-mon[27428]: osdmap e100: 8 total, 8 up, 8 in 2024-09-17T12:26:50.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:49 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2593520630' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1972581956"}]: dispatch 2024-09-17T12:26:50.070 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:49 smithi016 ceph-mon[23719]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:50.071 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:49 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2925162511' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2599403596"}]': finished 2024-09-17T12:26:50.071 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:49 smithi016 ceph-mon[23719]: osdmap e100: 8 total, 8 up, 8 in 2024-09-17T12:26:50.071 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:49 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2593520630' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1972581956"}]: dispatch 2024-09-17T12:26:50.071 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:49 smithi016 ceph-mon[34212]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:50.071 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:49 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2925162511' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2599403596"}]': finished 2024-09-17T12:26:50.071 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:49 smithi016 ceph-mon[34212]: osdmap e100: 8 total, 8 up, 8 in 2024-09-17T12:26:50.071 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:49 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2593520630' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1972581956"}]: dispatch 2024-09-17T12:26:51.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:51 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2593520630' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1972581956"}]': finished 2024-09-17T12:26:51.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:51 smithi196 ceph-mon[27428]: osdmap e101: 8 total, 8 up, 8 in 2024-09-17T12:26:51.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:51 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/1088212322' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1687402001"}]: dispatch 2024-09-17T12:26:51.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:51 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1687402001"}]: dispatch 2024-09-17T12:26:51.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:51 smithi196 ceph-mon[27428]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:51.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:51 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:51.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:51 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:51.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:51 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:51.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:51 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:26:51.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:51 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:51.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:51 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:26:51.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:51 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:26:51.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:51 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:51.540 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2593520630' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1972581956"}]': finished 2024-09-17T12:26:51.540 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[34212]: osdmap e101: 8 total, 8 up, 8 in 2024-09-17T12:26:51.540 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/1088212322' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1687402001"}]: dispatch 2024-09-17T12:26:51.541 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1687402001"}]: dispatch 2024-09-17T12:26:51.541 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[34212]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:51.541 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:51.541 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:51.541 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:51.541 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:26:51.542 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:51.542 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:26:51.542 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:26:51.542 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:51.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2593520630' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1972581956"}]': finished 2024-09-17T12:26:51.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[23719]: osdmap e101: 8 total, 8 up, 8 in 2024-09-17T12:26:51.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/1088212322' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1687402001"}]: dispatch 2024-09-17T12:26:51.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1687402001"}]: dispatch 2024-09-17T12:26:51.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[23719]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:26:51.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:51.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:51.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:51.544 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:26:51.544 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' 2024-09-17T12:26:51.544 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:26:51.544 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T12:26:51.544 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:51 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:26:52.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:52 smithi196 ceph-mon[27428]: Reconfiguring iscsi.foo.smithi016.qaboca (dependencies changed)... 2024-09-17T12:26:52.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:52 smithi196 ceph-mon[27428]: Reconfiguring daemon iscsi.foo.smithi016.qaboca on smithi016 2024-09-17T12:26:52.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:52 smithi196 ceph-mon[27428]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1687402001"}]': finished 2024-09-17T12:26:52.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:52 smithi196 ceph-mon[27428]: osdmap e102: 8 total, 8 up, 8 in 2024-09-17T12:26:52.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:52 smithi016 ceph-mon[23719]: Reconfiguring iscsi.foo.smithi016.qaboca (dependencies changed)... 2024-09-17T12:26:52.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:52 smithi016 ceph-mon[23719]: Reconfiguring daemon iscsi.foo.smithi016.qaboca on smithi016 2024-09-17T12:26:52.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:52 smithi016 ceph-mon[23719]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1687402001"}]': finished 2024-09-17T12:26:52.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:52 smithi016 ceph-mon[23719]: osdmap e102: 8 total, 8 up, 8 in 2024-09-17T12:26:52.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:52 smithi016 ceph-mon[34212]: Reconfiguring iscsi.foo.smithi016.qaboca (dependencies changed)... 2024-09-17T12:26:52.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:52 smithi016 ceph-mon[34212]: Reconfiguring daemon iscsi.foo.smithi016.qaboca on smithi016 2024-09-17T12:26:52.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:52 smithi016 ceph-mon[34212]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1687402001"}]': finished 2024-09-17T12:26:52.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:52 smithi016 ceph-mon[34212]: osdmap e102: 8 total, 8 up, 8 in 2024-09-17T12:26:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:53 smithi016 ceph-mon[23719]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:53.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:53 smithi016 ceph-mon[34212]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:53.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:53 smithi196 ceph-mon[27428]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:54.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:26:54] ENGINE Bus STOPPING 2024-09-17T12:26:54.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:26:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T12:26:54.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:26:54] ENGINE Bus STOPPED 2024-09-17T12:26:54.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:54.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T12:26:54.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T12:26:54.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: Activating manager daemon y 2024-09-17T12:26:54.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T12:26:54.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.24895 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T12:26:54.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: mgrmap e30: y(active, starting, since 0.00772389s) 2024-09-17T12:26:54.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:26:54.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:26:54.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:26:54.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:26:54.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:26:54.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:26:54.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:26:54.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:26:54.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:26:54.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:26:54.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:26:54.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:26:54.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:26:54.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:26:54.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:26:54.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T12:26:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T12:26:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: Activating manager daemon y 2024-09-17T12:26:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T12:26:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.24895 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T12:26:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: mgrmap e30: y(active, starting, since 0.00772389s) 2024-09-17T12:26:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:26:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:26:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:26:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:26:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:26:54.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:26:54.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:26:54.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:26:54.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:26:54.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:26:54.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:26:54.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:26:54.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:26:54.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:26:54.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:54 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:26:55.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.24895 172.21.15.196:0/1753877315' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T12:26:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T12:26:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: Activating manager daemon y 2024-09-17T12:26:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T12:26:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.24895 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T12:26:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: mgrmap e30: y(active, starting, since 0.00772389s) 2024-09-17T12:26:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:26:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:26:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:26:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:26:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:26:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:26:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:26:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:26:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:26:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:26:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:26:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:26:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:26:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:26:55.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:26:55.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: 2024-09-17T12:26:54.557+0000 7fa97d3d3640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.qaboca ... 2024-09-17T12:26:55.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Creating ceph-iscsi config... 2024-09-17T12:26:55.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/iscsi-gateway.cfg 2024-09-17T12:26:55.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/tcmu-runner-entrypoint.sh 2024-09-17T12:26:55.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Non-zero exit code 1 from systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca 2024-09-17T12:26:55.029 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: systemctl: stderr Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:55.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: systemctl: stderr See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:55.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Traceback (most recent call last): 2024-09-17T12:26:55.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T12:26:55.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: return _run_code(code, main_globals, None, 2024-09-17T12:26:55.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T12:26:55.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: exec(code, run_globals) 2024-09-17T12:26:55.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5628, in 2024-09-17T12:26:55.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5616, in main 2024-09-17T12:26:55.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-17T12:26:55.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-17T12:26:55.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-17T12:26:55.030 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-17T12:26:55.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T12:26:55.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: RuntimeError: Failed command: systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca: Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:55.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:55.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Traceback (most recent call last): 2024-09-17T12:26:55.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-17T12:26:55.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T12:26:55.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2557, in _daemon_action 2024-09-17T12:26:55.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: return self.wait_async( 2024-09-17T12:26:55.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-17T12:26:55.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: return self.event_loop.get_result(coro, timeout) 2024-09-17T12:26:55.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T12:26:55.031 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: return future.result(timeout) 2024-09-17T12:26:55.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T12:26:55.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: return self.__get_result() 2024-09-17T12:26:55.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T12:26:55.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: raise self._exception 2024-09-17T12:26:55.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-17T12:26:55.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: out, err, code = await self._run_cephadm( 2024-09-17T12:26:55.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-17T12:26:55.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: raise OrchestratorError( 2024-09-17T12:26:55.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.qaboca ... 2024-09-17T12:26:55.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Creating ceph-iscsi config... 2024-09-17T12:26:55.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/iscsi-gateway.cfg 2024-09-17T12:26:55.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/tcmu-runner-entrypoint.sh 2024-09-17T12:26:55.032 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Non-zero exit code 1 from systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: systemctl: stderr Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: systemctl: stderr See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: Traceback (most recent call last): 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: return _run_code(code, main_globals, None, 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: exec(code, run_globals) 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5628, in 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5616, in main 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-17T12:26:55.033 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: File "/tmp/tmpw5qw9iga.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T12:26:55.034 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: RuntimeError: Failed command: systemctl restart ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca: Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:26:55.034 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:26:55.034 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: ignoring --setuser ceph since I am not root 2024-09-17T12:26:55.034 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: ignoring --setgroup ceph since I am not root 2024-09-17T12:26:55.034 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:54 smithi196 ceph-mgr[62425]: -- 172.21.15.196:0/435097715 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aa1a7650e0 con 0x55aa1a742800 2024-09-17T12:26:55.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-mon[23719]: Manager daemon y is now available 2024-09-17T12:26:55.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:55.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:55.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:26:55.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:26:55.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:26:55.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-mon[34212]: Manager daemon y is now available 2024-09-17T12:26:55.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:55.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:55.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:26:55.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:26:55.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:26:55.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:26:55] ENGINE Bus STARTING 2024-09-17T12:26:55.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:26:55] ENGINE Serving on http://:::9283 2024-09-17T12:26:55.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:55 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:26:55] ENGINE Bus STARTED 2024-09-17T12:26:56.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:55 smithi196 ceph-mon[27428]: Manager daemon y is now available 2024-09-17T12:26:56.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:55 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:26:56.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:55 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:26:56.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:55 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:26:56.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:55 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:26:56.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:55 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:26:56.412 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:26:56.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:56 smithi196 ceph-mon[27428]: mgrmap e31: y(active, since 1.06353s) 2024-09-17T12:26:56.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:56 smithi196 ceph-mon[27428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:26:56.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:56 smithi196 ceph-mon[27428]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:56.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:56 smithi196 ceph-mon[27428]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:56.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:56 smithi196 ceph-mon[27428]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:56.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-mon[23719]: mgrmap e31: y(active, since 1.06353s) 2024-09-17T12:26:56.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-mon[23719]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:26:56.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-mon[23719]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:56.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-mon[23719]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:56.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-mon[23719]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:56.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-mon[34212]: mgrmap e31: y(active, since 1.06353s) 2024-09-17T12:26:56.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-mon[34212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:26:56.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-mon[34212]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:56.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-mon[34212]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:56.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-mon[34212]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:56.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:26:56.852+0000 7f2919975640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:12:26:56] ENGINE Error in HTTPServer.serve 2024-09-17T12:26:56.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: Traceback (most recent call last): 2024-09-17T12:26:56.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:26:56.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: self._connections.run(self.expiration_interval) 2024-09-17T12:26:56.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:26:56.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: self._run(expiration_interval) 2024-09-17T12:26:56.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:26:56.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:26:56.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:26:56.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:26:56.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:26:56.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: s = self.context.wrap_socket( 2024-09-17T12:26:56.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:26:56.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: return self.sslsocket_class._create( 2024-09-17T12:26:56.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:26:56.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: self.do_handshake() 2024-09-17T12:26:56.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:26:56.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: self._sslobj.do_handshake() 2024-09-17T12:26:56.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:26:56.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:26:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:26:57.023 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:26:57.023 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (5m) 10s ago 5m 26.0M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:26:57.023 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (5m) 10s ago 5m 85.6M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:26:57.023 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (14s) 10s ago 4m - - 3.8 980a0bfcd684 d459aaa1955d 2024-09-17T12:26:57.023 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (59s) 10s ago 9m 512M - 19.3.0-4937-g74984932 6c1fcab85602 7ba30e0f95c9 2024-09-17T12:26:57.024 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (15s) 10s ago 11m 64.5M - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:26:57.024 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (11m) 10s ago 11m 57.4M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 5d21f110d6fc 2024-09-17T12:26:57.024 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (10m) 10s ago 10m 47.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 262ef622777e 2024-09-17T12:26:57.024 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (10m) 10s ago 9m 50.9M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 950b72634364 2024-09-17T12:26:57.024 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (5m) 10s ago 5m 15.7M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:26:57.024 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (5m) 10s ago 5m 15.7M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:26:57.024 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (9m) 10s ago 9m 61.5M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 e30aaffd3b5f 2024-09-17T12:26:57.024 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (9m) 10s ago 9m 53.7M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 3ad94c30c9c9 2024-09-17T12:26:57.024 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (8m) 10s ago 8m 54.4M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 1fa310695876 2024-09-17T12:26:57.024 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (8m) 10s ago 8m 59.1M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 4a4f8eeaff13 2024-09-17T12:26:57.024 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (7m) 10s ago 7m 54.3M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 afa20056b398 2024-09-17T12:26:57.024 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (7m) 10s ago 7m 50.1M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 f0dcabb8902f 2024-09-17T12:26:57.024 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (7m) 10s ago 7m 57.9M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:26:57.025 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (6m) 10s ago 6m 60.7M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:26:57.025 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (20s) 10s ago 6m 34.4M - 2.43.0 a07b618ecd1d 65e201311504 2024-09-17T12:26:57.025 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (4m) 10s ago 4m 77.9M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:26:57.025 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (4m) 10s ago 4m 78.6M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:26:57.025 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (4m) 10s ago 4m 79.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:26:57.025 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (4m) 10s ago 4m 77.6M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:26:57.385 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:26:57.386 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:26:57.386 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T12:26:57.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:26:57.386 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:26:57.386 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 1 2024-09-17T12:26:57.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:26:57.386 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:26:57.386 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T12:26:57.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:26:57.386 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:26:57.387 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:26:57.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:26:57.387 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:26:57.387 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 15, 2024-09-17T12:26:57.387 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 1 2024-09-17T12:26:57.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:26:57.387 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:26:57.697 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:26:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:26:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:26:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:26:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:26:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:26:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:26:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "2/23 daemons upgraded", 2024-09-17T12:26:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "", 2024-09-17T12:26:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:26:57.698 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:26:57.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[23719]: [17/Sep/2024:12:26:56] ENGINE Bus STARTING 2024-09-17T12:26:57.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[23719]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:57.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[23719]: [17/Sep/2024:12:26:56] ENGINE Serving on http://172.21.15.16:8765 2024-09-17T12:26:57.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[23719]: [17/Sep/2024:12:26:56] ENGINE Serving on https://172.21.15.16:7150 2024-09-17T12:26:57.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[23719]: [17/Sep/2024:12:26:56] ENGINE Bus STARTED 2024-09-17T12:26:57.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[23719]: [17/Sep/2024:12:26:56] ENGINE Error in HTTPServer.serve 2024-09-17T12:26:57.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:26:57.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:26:57.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:26:57.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:26:57.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-09-17T12:26:57.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:26:57.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:26:57.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:26:57.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:26:57.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:26:57.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-09-17T12:26:57.942 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:26:57.943 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-09-17T12:26:57.943 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:26:57.943 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-09-17T12:26:57.943 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:26:57.943 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-09-17T12:26:57.943 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:26:57.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[23719]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:57.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3975516661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:26:57.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[34212]: [17/Sep/2024:12:26:56] ENGINE Bus STARTING 2024-09-17T12:26:57.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[34212]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:57.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[34212]: [17/Sep/2024:12:26:56] ENGINE Serving on http://172.21.15.16:8765 2024-09-17T12:26:57.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[34212]: [17/Sep/2024:12:26:56] ENGINE Serving on https://172.21.15.16:7150 2024-09-17T12:26:57.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[34212]: [17/Sep/2024:12:26:56] ENGINE Bus STARTED 2024-09-17T12:26:57.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[34212]: [17/Sep/2024:12:26:56] ENGINE Error in HTTPServer.serve 2024-09-17T12:26:57.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:26:57.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:26:57.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:26:57.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:26:57.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: self._run(expiration_interval) 2024-09-17T12:26:57.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:26:57.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:26:57.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:26:57.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:26:57.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:26:57.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: s = self.context.wrap_socket( 2024-09-17T12:26:57.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:26:57.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: return self.sslsocket_class._create( 2024-09-17T12:26:57.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:26:57.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: self.do_handshake() 2024-09-17T12:26:57.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:26:57.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: self._sslobj.do_handshake() 2024-09-17T12:26:57.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:26:57.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[34212]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:57.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:57 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3975516661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:26:58.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:57 smithi196 ceph-mon[27428]: [17/Sep/2024:12:26:56] ENGINE Bus STARTING 2024-09-17T12:26:58.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:57 smithi196 ceph-mon[27428]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:58.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:57 smithi196 ceph-mon[27428]: [17/Sep/2024:12:26:56] ENGINE Serving on http://172.21.15.16:8765 2024-09-17T12:26:58.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:57 smithi196 ceph-mon[27428]: [17/Sep/2024:12:26:56] ENGINE Serving on https://172.21.15.16:7150 2024-09-17T12:26:58.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:57 smithi196 ceph-mon[27428]: [17/Sep/2024:12:26:56] ENGINE Bus STARTED 2024-09-17T12:26:58.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:57 smithi196 ceph-mon[27428]: [17/Sep/2024:12:26:56] ENGINE Error in HTTPServer.serve 2024-09-17T12:26:58.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-17T12:26:58.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:26:58.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:26:58.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:26:58.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._run(expiration_interval) 2024-09-17T12:26:58.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:26:58.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:26:58.027 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:26:58.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:26:58.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:26:58.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: s = self.context.wrap_socket( 2024-09-17T12:26:58.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:26:58.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.sslsocket_class._create( 2024-09-17T12:26:58.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:26:58.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.do_handshake() 2024-09-17T12:26:58.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:26:58.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._sslobj.do_handshake() 2024-09-17T12:26:58.028 INFO:journalctl@ceph.mon.b.smithi196.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:26:58.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:57 smithi196 ceph-mon[27428]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:58.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:57 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3975516661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:26:58.062 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-09-17T12:26:58.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:58 smithi016 ceph-mon[23719]: mgrmap e32: y(active, since 3s) 2024-09-17T12:26:58.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:58 smithi016 ceph-mon[23719]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:58.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:58 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/3301790225' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:26:58.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:26:58 smithi016 ceph-mon[23719]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:58.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:58 smithi016 ceph-mon[34212]: mgrmap e32: y(active, since 3s) 2024-09-17T12:26:58.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:58 smithi016 ceph-mon[34212]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:58.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:58 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3301790225' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:26:58.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:26:58 smithi016 ceph-mon[34212]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:26:58.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:58 smithi196 ceph-mon[27428]: mgrmap e32: y(active, since 3s) 2024-09-17T12:26:58.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:58 smithi196 ceph-mon[27428]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:26:58.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:58 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3301790225' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:26:58.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:26:58 smithi196 ceph-mon[27428]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:27:00.276 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:59 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: [17/Sep/2024:12:26:59] ENGINE Bus STARTING 2024-09-17T12:27:00.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:59 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: CherryPy Checker: 2024-09-17T12:27:00.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:59 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: The Application mounted at '' has an empty config. 2024-09-17T12:27:00.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:59 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: 2024-09-17T12:27:00.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:59 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: [17/Sep/2024:12:26:59] ENGINE Serving on http://:::9283 2024-09-17T12:27:00.277 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:26:59 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: [17/Sep/2024:12:26:59] ENGINE Bus STARTED 2024-09-17T12:27:00.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:00.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:00.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[23719]: mgrmap e33: y(active, since 5s) 2024-09-17T12:27:00.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[23719]: Standby manager daemon x started 2024-09-17T12:27:00.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/1278323599' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:27:00.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/1278323599' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:27:00.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/1278323599' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:27:00.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/1278323599' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:27:00.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:00.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:00.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:00.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:00.816 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:00.816 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:00.816 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[34212]: mgrmap e33: y(active, since 5s) 2024-09-17T12:27:00.816 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[34212]: Standby manager daemon x started 2024-09-17T12:27:00.816 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/1278323599' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:27:00.816 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/1278323599' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:27:00.817 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/1278323599' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:27:00.817 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/1278323599' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:27:00.817 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:00.817 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:00.817 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:00.817 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:00 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:01.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:00 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:01.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:00 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:00 smithi196 ceph-mon[27428]: mgrmap e33: y(active, since 5s) 2024-09-17T12:27:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:00 smithi196 ceph-mon[27428]: Standby manager daemon x started 2024-09-17T12:27:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:00 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/1278323599' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:27:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:00 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/1278323599' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:27:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:00 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/1278323599' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:27:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:00 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/1278323599' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:27:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:00 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:00 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:00 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:00 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:01.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:01 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:01.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:01 smithi016 ceph-mon[23719]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:27:01.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:01 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:01.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:01 smithi016 ceph-mon[23719]: mgrmap e34: y(active, since 6s), standbys: x 2024-09-17T12:27:01.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:01 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:27:01.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:01 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:01.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:01 smithi016 ceph-mon[34212]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:27:01.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:01 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:01.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:01 smithi016 ceph-mon[34212]: mgrmap e34: y(active, since 6s), standbys: x 2024-09-17T12:27:01.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:01 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:27:02.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:01 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:02.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:01 smithi196 ceph-mon[27428]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:27:02.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:01 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:02.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:01 smithi196 ceph-mon[27428]: mgrmap e34: y(active, since 6s), standbys: x 2024-09-17T12:27:02.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:01 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:27:02.809 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:02 smithi016 ceph-mon[34212]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T12:27:02.810 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:02 smithi016 ceph-mon[23719]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T12:27:03.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:02 smithi196 ceph-mon[27428]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T12:27:04.518 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.518 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.518 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:04.518 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:27:04.518 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:27:04.518 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:27:04.518 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:27:04.518 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:27:04.519 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:27:04.519 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:27:04.519 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:27:04.519 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:27:04.519 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.519 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.519 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.519 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.519 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:04 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:04.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:27:04.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:27:04.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:27:04.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:27:04.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:27:04.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:27:04.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:27:04.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:27:04.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:27:04.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:04.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:27:04.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:27:04.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:27:04.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:27:04.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:27:04.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:27:04.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:27:04.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:27:04.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:27:04.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:04.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:04 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:05.424 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:05 smithi016 ceph-mon[23719]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-17T12:27:05.424 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:05 smithi016 ceph-mon[23719]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T12:27:05.424 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:05 smithi016 ceph-mon[23719]: Reconfiguring daemon prometheus.a on smithi196 2024-09-17T12:27:05.424 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:05 smithi016 ceph-mon[23719]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T12:27:05.425 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:05 smithi016 ceph-mon[34212]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-17T12:27:05.425 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:05 smithi016 ceph-mon[34212]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T12:27:05.425 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:05 smithi016 ceph-mon[34212]: Reconfiguring daemon prometheus.a on smithi196 2024-09-17T12:27:05.425 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:05 smithi016 ceph-mon[34212]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T12:27:05.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:05 smithi196 ceph-mon[27428]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-17T12:27:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:05 smithi196 ceph-mon[27428]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T12:27:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:05 smithi196 ceph-mon[27428]: Reconfiguring daemon prometheus.a on smithi196 2024-09-17T12:27:05.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:05 smithi196 ceph-mon[27428]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T12:27:07.026 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:06 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[62421]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:27:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-17T12:27:07.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-mon[27428]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-17T12:27:07.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2608806104' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T12:27:07.505 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 systemd[1]: Stopping Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:27:07.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:07 smithi016 ceph-mon[34212]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-17T12:27:07.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:07 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2608806104' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T12:27:07.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:07 smithi016 ceph-mon[23719]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-17T12:27:07.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:07 smithi016 ceph-mon[23719]: from='client.? 172.21.15.16:0/2608806104' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T12:27:07.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:27:07.503Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T12:27:07.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:27:07.503Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-17T12:27:07.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:27:07.503Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-17T12:27:07.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:27:07.503Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-17T12:27:07.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:27:07.503Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T12:27:07.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:27:07.503Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-17T12:27:07.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:27:07.504Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T12:27:07.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:27:07.504Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-17T12:27:07.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:27:07.504Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-17T12:27:07.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:27:07.504Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T12:27:07.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:27:07.504Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-17T12:27:07.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[70695]: ts=2024-09-17T12:27:07.504Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-17T12:27:07.778 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 podman[72643]: 2024-09-17 12:27:07.514049547 +0000 UTC m=+0.117726602 container died 65e20131150468bd6e7308983c45ea00fa892076b471c4c7b1c009f004ff8ce2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:27:08.122 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 podman[72643]: 2024-09-17 12:27:07.869937514 +0000 UTC m=+0.473614558 container cleanup 65e20131150468bd6e7308983c45ea00fa892076b471c4c7b1c009f004ff8ce2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:27:08.122 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:07 smithi196 bash[72643]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T12:27:08.499 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 podman[72655]: 2024-09-17 12:27:08.121361342 +0000 UTC m=+0.609049719 container remove 65e20131150468bd6e7308983c45ea00fa892076b471c4c7b1c009f004ff8ce2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:27:08.499 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T12:27:08.499 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 systemd[1]: Stopped Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:27:08.499 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.048s CPU time. 2024-09-17T12:27:08.499 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 systemd[1]: Starting Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:27:08.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 podman[72746]: 2024-09-17 12:27:08.401536971 +0000 UTC m=+0.017457532 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-17T12:27:08.777 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 podman[72746]: 2024-09-17 12:27:08.530676614 +0000 UTC m=+0.146597206 container create fd41eb792eebd41bcc38c2b7e0a6f442119588dde89e911d5188c11d91b2239d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:27:09.033 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 podman[72746]: 2024-09-17 12:27:08.833155416 +0000 UTC m=+0.449075970 container init fd41eb792eebd41bcc38c2b7e0a6f442119588dde89e911d5188c11d91b2239d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:27:09.034 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 podman[72746]: 2024-09-17 12:27:08.836194459 +0000 UTC m=+0.452115003 container start fd41eb792eebd41bcc38c2b7e0a6f442119588dde89e911d5188c11d91b2239d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:27:09.034 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.859Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-17T12:27:09.034 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.859Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-17T12:27:09.034 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.859Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi196 (none))" 2024-09-17T12:27:09.034 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.859Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T12:27:09.034 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.859Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T12:27:09.034 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.860Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-17T12:27:09.034 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.861Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-17T12:27:09.034 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.862Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-17T12:27:09.034 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.862Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-17T12:27:09.035 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.864Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-17T12:27:09.035 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.864Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.664µs 2024-09-17T12:27:09.035 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.864Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-17T12:27:09.035 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.866Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-17T12:27:09.035 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.875Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-17T12:27:09.035 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.876Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-17T12:27:09.035 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.876Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-17T12:27:09.035 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.876Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.413µs wal_replay_duration=11.700499ms wbl_replay_duration=110ns total_replay_duration=11.742394ms 2024-09-17T12:27:09.035 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.878Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-17T12:27:09.035 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.878Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-17T12:27:09.035 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.878Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T12:27:09.035 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.897Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.605442ms db_storage=751ns remote_storage=1.274µs web_handler=349ns query_engine=489ns scrape=249.84µs scrape_sd=125.182µs notify=10.098µs notify_sd=20.679µs rules=18.862483ms tracing=5.376µs 2024-09-17T12:27:09.035 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.897Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-17T12:27:09.036 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:27:08.897Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-17T12:27:09.036 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 bash[72746]: fd41eb792eebd41bcc38c2b7e0a6f442119588dde89e911d5188c11d91b2239d 2024-09-17T12:27:09.036 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:27:08 smithi196 systemd[1]: Started Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:27:09.370 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:09 smithi196 ceph-mon[27428]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:09.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:09 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:09.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:09 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:09.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:09 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:27:09.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:09 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:09.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:09 smithi016 ceph-mon[34212]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:09.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:09 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:09.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:09 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:09.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:09 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:27:09.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:09 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:09.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:09 smithi016 ceph-mon[23719]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:09.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:09 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:09.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:09 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:09.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:09 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:27:09.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:09 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:10.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:10 smithi196 ceph-mon[27428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:27:10.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:10 smithi016 ceph-mon[34212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:27:10.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:10 smithi016 ceph-mon[23719]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T12:27:11.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:11 smithi196 ceph-mon[27428]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:11 smithi196 ceph-mon[27428]: Upgrade: Updating mgr.x 2024-09-17T12:27:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:11 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:27:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:11 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:27:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:11 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:11 smithi196 ceph-mon[27428]: Deploying daemon mgr.x on smithi196 2024-09-17T12:27:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:11 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:11 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:27:11.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[23719]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:11.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[23719]: Upgrade: Updating mgr.x 2024-09-17T12:27:11.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:27:11.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:27:11.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:11.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[23719]: Deploying daemon mgr.x on smithi196 2024-09-17T12:27:11.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:11.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:27:11.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[34212]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:11.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[34212]: Upgrade: Updating mgr.x 2024-09-17T12:27:11.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T12:27:11.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T12:27:11.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:11.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[34212]: Deploying daemon mgr.x on smithi196 2024-09-17T12:27:11.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:11.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:11 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:27:12.267 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:11 smithi196 systemd[1]: Stopping Ceph mgr.x for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:27:12.519 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:12 smithi196 podman[73017]: 2024-09-17 12:27:12.265690267 +0000 UTC m=+0.151970647 container died 7ba30e0f95c9f92915e51f5688cb04efc27d73a8be8509f36ed09d565fd0d3c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, RELEASE=main-7498493, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-17T12:27:12.776 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:12 smithi196 podman[73017]: 2024-09-17 12:27:12.600693068 +0000 UTC m=+0.486973439 container cleanup 7ba30e0f95c9f92915e51f5688cb04efc27d73a8be8509f36ed09d565fd0d3c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-7498493, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-17T12:27:12.777 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:12 smithi196 bash[73017]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x 2024-09-17T12:27:13.086 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:12 smithi196 podman[73029]: 2024-09-17 12:27:12.835224195 +0000 UTC m=+0.565128815 container remove 7ba30e0f95c9f92915e51f5688cb04efc27d73a8be8509f36ed09d565fd0d3c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-7498493, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:27:13.086 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:12 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-17T12:27:13.087 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:13 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-17T12:27:13.087 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:13 smithi196 systemd[1]: Stopped Ceph mgr.x for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:27:13.087 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:13 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.x.service: Consumed 17.119s CPU time. 2024-09-17T12:27:13.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:13 smithi196 ceph-mon[27428]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:13.475 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:13 smithi196 systemd[1]: Starting Ceph mgr.x for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:27:13.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:13 smithi016 ceph-mon[23719]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:13.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:13 smithi016 ceph-mon[34212]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:13.773 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:13 smithi196 podman[73158]: 2024-09-17 12:27:13.377078918 +0000 UTC m=+0.016168684 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:27:13.774 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:13 smithi196 podman[73158]: 2024-09-17 12:27:13.511396563 +0000 UTC m=+0.150486363 container create 89153031794c53c564f534571bfcdf43d71f84cd952570bb2e7f709442c328c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-7498493, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-17T12:27:13.774 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:13 smithi196 podman[73158]: 2024-09-17 12:27:13.672228362 +0000 UTC m=+0.311318124 container init 89153031794c53c564f534571bfcdf43d71f84cd952570bb2e7f709442c328c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-7498493, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:27:13.774 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:13 smithi196 podman[73158]: 2024-09-17 12:27:13.674946276 +0000 UTC m=+0.314036034 container start 89153031794c53c564f534571bfcdf43d71f84cd952570bb2e7f709442c328c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-7498493, GIT_BRANCH=HEAD) 2024-09-17T12:27:13.774 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:13 smithi196 ceph-mgr[73177]: -- 172.21.15.196:0/222308302 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563038cf10e0 con 0x563038ccf400 2024-09-17T12:27:14.026 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:13 smithi196 bash[73158]: 89153031794c53c564f534571bfcdf43d71f84cd952570bb2e7f709442c328c9 2024-09-17T12:27:14.026 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:13 smithi196 systemd[1]: Started Ceph mgr.x for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:27:15.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:14 smithi016 ceph-mon[23719]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 459 B/s rd, 0 op/s 2024-09-17T12:27:15.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:14 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:15.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:14 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:15.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:14 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:15.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:14 smithi016 ceph-mon[34212]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 459 B/s rd, 0 op/s 2024-09-17T12:27:15.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:14 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:15.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:14 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:15.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:14 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:15.208 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:14 smithi196 ceph-mon[27428]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 459 B/s rd, 0 op/s 2024-09-17T12:27:15.208 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:14 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:15.208 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:14 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:15.208 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:14 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:16.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:27:16] "GET /metrics HTTP/1.1" 200 38509 "" "Prometheus/2.43.0" 2024-09-17T12:27:17.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:16 smithi196 ceph-mon[27428]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:27:17.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:16 smithi196 ceph-mon[27428]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:17.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:16 smithi016 ceph-mon[23719]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:27:17.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:16 smithi016 ceph-mon[23719]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:17.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:16 smithi016 ceph-mon[34212]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:27:17.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:16 smithi016 ceph-mon[34212]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:19.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:18 smithi196 ceph-mon[27428]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:27:19.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:18 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:19.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:18 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:19.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:18 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:19.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:18 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:19.142 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:19 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: [17/Sep/2024:12:27:19] ENGINE Bus STARTING 2024-09-17T12:27:19.142 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:19 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: CherryPy Checker: 2024-09-17T12:27:19.142 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:19 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: The Application mounted at '' has an empty config. 2024-09-17T12:27:19.142 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:19 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: 2024-09-17T12:27:19.142 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:19 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: [17/Sep/2024:12:27:19] ENGINE Serving on http://:::9283 2024-09-17T12:27:19.142 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:19 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: [17/Sep/2024:12:27:19] ENGINE Bus STARTED 2024-09-17T12:27:19.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:18 smithi016 ceph-mon[23719]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:27:19.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:18 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:19.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:18 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:19.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:18 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:19.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:18 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:19.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:18 smithi016 ceph-mon[34212]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:27:19.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:18 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:19.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:18 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:19.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:18 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:19.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:18 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:20.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:19 smithi016 ceph-mon[23719]: Standby manager daemon x restarted 2024-09-17T12:27:20.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:19 smithi016 ceph-mon[23719]: Standby manager daemon x started 2024-09-17T12:27:20.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:19 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/3680200258' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:27:20.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:19 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/3680200258' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:27:20.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:19 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/3680200258' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:27:20.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:19 smithi016 ceph-mon[23719]: from='mgr.? 172.21.15.196:0/3680200258' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:27:20.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:19 smithi016 ceph-mon[34212]: Standby manager daemon x restarted 2024-09-17T12:27:20.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:19 smithi016 ceph-mon[34212]: Standby manager daemon x started 2024-09-17T12:27:20.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:19 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/3680200258' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:27:20.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:19 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/3680200258' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:27:20.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:19 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/3680200258' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:27:20.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:19 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/3680200258' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:27:20.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:19 smithi196 ceph-mon[27428]: Standby manager daemon x restarted 2024-09-17T12:27:20.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:19 smithi196 ceph-mon[27428]: Standby manager daemon x started 2024-09-17T12:27:20.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:19 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/3680200258' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:27:20.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:19 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/3680200258' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:27:20.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:19 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/3680200258' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:27:20.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:19 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/3680200258' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:27:21.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:20 smithi016 ceph-mon[23719]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:27:21.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:20 smithi016 ceph-mon[23719]: mgrmap e35: y(active, since 25s), standbys: x 2024-09-17T12:27:21.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:20 smithi016 ceph-mon[34212]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:27:21.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:20 smithi016 ceph-mon[34212]: mgrmap e35: y(active, since 25s), standbys: x 2024-09-17T12:27:21.249 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:20 smithi196 ceph-mon[27428]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:27:21.250 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:20 smithi196 ceph-mon[27428]: mgrmap e35: y(active, since 25s), standbys: x 2024-09-17T12:27:22.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:22.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:22.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:22.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:27:22.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:22.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:22.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:27:22.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:22.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T12:27:22.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-17T12:27:22.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T12:27:22.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-17T12:27:22.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T12:27:22.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-17T12:27:22.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:22.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:22.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:22.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:27:22.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:22.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:22.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:27:22.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:22.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T12:27:22.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-17T12:27:22.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T12:27:22.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-17T12:27:22.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T12:27:22.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:22 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-17T12:27:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:27:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:27:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T12:27:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-17T12:27:23.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T12:27:23.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-17T12:27:23.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T12:27:23.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:22 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-17T12:27:23.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[23719]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T12:27:23.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[23719]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T12:27:23.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[23719]: Upgrade: Setting container_image for all mgr 2024-09-17T12:27:23.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[23719]: Upgrade: It appears safe to stop mon.a 2024-09-17T12:27:23.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:23.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:27:23.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:27:23.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[23719]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:23.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[34212]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T12:27:23.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[34212]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T12:27:23.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[34212]: Upgrade: Setting container_image for all mgr 2024-09-17T12:27:23.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[34212]: Upgrade: It appears safe to stop mon.a 2024-09-17T12:27:23.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:23.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:27:23.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:27:23.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:23 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:24.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:23 smithi196 ceph-mon[27428]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T12:27:24.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:23 smithi196 ceph-mon[27428]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T12:27:24.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:23 smithi196 ceph-mon[27428]: Upgrade: Setting container_image for all mgr 2024-09-17T12:27:24.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:23 smithi196 ceph-mon[27428]: Upgrade: It appears safe to stop mon.a 2024-09-17T12:27:24.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:23 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' 2024-09-17T12:27:24.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:23 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:27:24.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:23 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:27:24.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:23 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:24.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:24 smithi016 ceph-mon[23719]: Upgrade: Updating mon.a 2024-09-17T12:27:24.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:24 smithi016 ceph-mon[23719]: Deploying daemon mon.a on smithi016 2024-09-17T12:27:24.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:24 smithi016 ceph-mon[34212]: Upgrade: Updating mon.a 2024-09-17T12:27:24.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:24 smithi016 ceph-mon[34212]: Deploying daemon mon.a on smithi016 2024-09-17T12:27:25.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:24 smithi016 systemd[1]: Stopping Ceph mon.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:27:25.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:24 smithi196 ceph-mon[27428]: Upgrade: Updating mon.a 2024-09-17T12:27:25.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:24 smithi196 ceph-mon[27428]: Deploying daemon mon.a on smithi016 2024-09-17T12:27:25.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:25 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a[23715]: 2024-09-17T12:27:25.009+0000 7f6a7e3ae640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:27:25.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:25 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a[23715]: 2024-09-17T12:27:25.009+0000 7f6a7e3ae640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-17T12:27:25.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:25 smithi016 podman[87718]: 2024-09-17 12:27:25.102490349 +0000 UTC m=+0.201086400 container died 5d21f110d6fca319f092f5ad2a5980f2fbff43b6fbcc051eb597b4d37570e531 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:27:25.333 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:25 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:25] ENGINE Bus STOPPING 2024-09-17T12:27:25.333 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:25 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T12:27:25.333 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:25 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:25] ENGINE Bus STOPPED 2024-09-17T12:27:25.333 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:25 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:25] ENGINE Bus STARTING 2024-09-17T12:27:25.333 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:25 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:25] ENGINE Serving on http://:::9283 2024-09-17T12:27:25.333 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:25 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:25] ENGINE Bus STARTED 2024-09-17T12:27:25.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:25 smithi016 podman[87718]: 2024-09-17 12:27:25.552131374 +0000 UTC m=+0.650727427 container cleanup 5d21f110d6fca319f092f5ad2a5980f2fbff43b6fbcc051eb597b4d37570e531 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:27:25.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:25 smithi016 bash[87718]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a 2024-09-17T12:27:26.185 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:25 smithi016 podman[87730]: 2024-09-17 12:27:25.845399962 +0000 UTC m=+0.737017530 container remove 5d21f110d6fca319f092f5ad2a5980f2fbff43b6fbcc051eb597b4d37570e531 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-17T12:27:26.185 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:25 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-17T12:27:26.185 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:25 smithi016 systemd[1]: Stopped Ceph mon.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:27:26.185 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:25 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.a.service: Consumed 14.976s CPU time. 2024-09-17T12:27:26.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 systemd[1]: Starting Ceph mon.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:27:26.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 podman[87870]: 2024-09-17 12:27:26.300005596 +0000 UTC m=+0.016689939 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:27:26.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:27:26] "GET /metrics HTTP/1.1" 200 38506 "" "Prometheus/2.43.0" 2024-09-17T12:27:26.707 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 podman[87870]: 2024-09-17 12:27:26.456091201 +0000 UTC m=+0.172775604 container create 4a83b787c6059560782c44c0bbc408d98604596747a05d9ab1df11eeb2c274c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=main-7498493, GIT_CLEAN=True) 2024-09-17T12:27:26.707 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 podman[87870]: 2024-09-17 12:27:26.674494773 +0000 UTC m=+0.391179114 container init 4a83b787c6059560782c44c0bbc408d98604596747a05d9ab1df11eeb2c274c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-7498493) 2024-09-17T12:27:26.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 podman[87870]: 2024-09-17 12:27:26.677840081 +0000 UTC m=+0.394524421 container start 4a83b787c6059560782c44c0bbc408d98604596747a05d9ab1df11eeb2c274c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-7498493, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:27:26.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T12:27:26.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev), process ceph-mon, pid 2 2024-09-17T12:27:26.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: pidfile_write: ignore empty --pid-file 2024-09-17T12:27:26.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: load: jerasure load: lrc 2024-09-17T12:27:26.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: RocksDB version: 7.9.2 2024-09-17T12:27:26.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Git sha 0 2024-09-17T12:27:26.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Compile date 2024-09-15 20:52:36 2024-09-17T12:27:26.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: DB SUMMARY 2024-09-17T12:27:26.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: DB Session ID: LZI0UXYYEADWCEA4M1XL 2024-09-17T12:27:26.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: CURRENT file: CURRENT 2024-09-17T12:27:26.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T12:27:26.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1497 Bytes 2024-09-17T12:27:26.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000032.sst 2024-09-17T12:27:26.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000030.log size: 2081045 ; 2024-09-17T12:27:26.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.error_if_exists: 0 2024-09-17T12:27:26.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.create_if_missing: 0 2024-09-17T12:27:26.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.paranoid_checks: 1 2024-09-17T12:27:26.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T12:27:26.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T12:27:26.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T12:27:26.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.env: 0x55fe7d1d9120 2024-09-17T12:27:26.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T12:27:26.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.info_log: 0x55fe7ea7f700 2024-09-17T12:27:26.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T12:27:26.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.statistics: (nil) 2024-09-17T12:27:26.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.use_fsync: 0 2024-09-17T12:27:26.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_log_file_size: 0 2024-09-17T12:27:26.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T12:27:26.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T12:27:26.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T12:27:26.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T12:27:26.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.allow_fallocate: 1 2024-09-17T12:27:26.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T12:27:26.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T12:27:26.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.use_direct_reads: 0 2024-09-17T12:27:26.712 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T12:27:26.712 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T12:27:26.712 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.db_log_dir: 2024-09-17T12:27:26.712 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.wal_dir: 2024-09-17T12:27:26.712 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T12:27:26.712 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T12:27:26.712 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T12:27:26.712 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T12:27:26.712 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T12:27:26.712 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T12:27:26.712 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T12:27:26.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T12:27:26.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.write_buffer_manager: 0x55fe7ea821e0 2024-09-17T12:27:26.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T12:27:26.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T12:27:26.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T12:27:26.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T12:27:26.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T12:27:26.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T12:27:26.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T12:27:26.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T12:27:26.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.unordered_write: 0 2024-09-17T12:27:26.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T12:27:26.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T12:27:26.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T12:27:26.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T12:27:26.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.row_cache: None 2024-09-17T12:27:26.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.wal_filter: None 2024-09-17T12:27:26.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T12:27:26.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T12:27:26.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.two_write_queues: 0 2024-09-17T12:27:26.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T12:27:26.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.wal_compression: 0 2024-09-17T12:27:26.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.atomic_flush: 0 2024-09-17T12:27:26.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T12:27:26.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T12:27:26.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T12:27:26.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.log_readahead_size: 0 2024-09-17T12:27:26.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T12:27:26.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T12:27:26.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T12:27:26.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T12:27:26.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T12:27:26.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T12:27:26.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T12:27:26.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_background_jobs: 2 2024-09-17T12:27:26.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_background_compactions: -1 2024-09-17T12:27:26.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_subcompactions: 1 2024-09-17T12:27:26.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T12:27:26.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T12:27:26.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T12:27:26.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T12:27:26.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T12:27:26.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T12:27:26.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T12:27:26.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T12:27:26.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_open_files: -1 2024-09-17T12:27:26.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T12:27:26.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T12:27:26.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T12:27:26.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T12:27:26.717 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_background_flushes: -1 2024-09-17T12:27:26.717 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Compression algorithms supported: 2024-09-17T12:27:26.717 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: kZSTD supported: 0 2024-09-17T12:27:26.717 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: kXpressCompression supported: 0 2024-09-17T12:27:26.717 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: kBZip2Compression supported: 0 2024-09-17T12:27:26.717 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T12:27:26.717 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: kLZ4Compression supported: 1 2024-09-17T12:27:26.717 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: kZlibCompression supported: 1 2024-09-17T12:27:26.717 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: kSnappyCompression supported: 1 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.merge_operator: 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compaction_filter: None 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fe7ea7f320) 2024-09-17T12:27:26.718 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-09-17T12:27:26.719 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T12:27:26.719 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T12:27:26.719 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-09-17T12:27:26.719 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_type: 0 2024-09-17T12:27:26.719 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_index_type: 0 2024-09-17T12:27:26.719 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_shortening: 1 2024-09-17T12:27:26.719 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T12:27:26.719 INFO:journalctl@ceph.mon.a.smithi016.stdout: checksum: 4 2024-09-17T12:27:26.719 INFO:journalctl@ceph.mon.a.smithi016.stdout: no_block_cache: 0 2024-09-17T12:27:26.719 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache: 0x55fe7ea73610 2024-09-17T12:27:26.720 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-09-17T12:27:26.720 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_options: 2024-09-17T12:27:26.720 INFO:journalctl@ceph.mon.a.smithi016.stdout: capacity : 536870912 2024-09-17T12:27:26.720 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_shard_bits : 4 2024-09-17T12:27:26.720 INFO:journalctl@ceph.mon.a.smithi016.stdout: strict_capacity_limit : 0 2024-09-17T12:27:26.720 INFO:journalctl@ceph.mon.a.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-09-17T12:27:26.720 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_compressed: (nil) 2024-09-17T12:27:26.720 INFO:journalctl@ceph.mon.a.smithi016.stdout: persistent_cache: (nil) 2024-09-17T12:27:26.720 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size: 4096 2024-09-17T12:27:26.720 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size_deviation: 10 2024-09-17T12:27:26.720 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_restart_interval: 16 2024-09-17T12:27:26.720 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_block_restart_interval: 1 2024-09-17T12:27:26.721 INFO:journalctl@ceph.mon.a.smithi016.stdout: metadata_block_size: 4096 2024-09-17T12:27:26.721 INFO:journalctl@ceph.mon.a.smithi016.stdout: partition_filters: 0 2024-09-17T12:27:26.721 INFO:journalctl@ceph.mon.a.smithi016.stdout: use_delta_encoding: 1 2024-09-17T12:27:26.721 INFO:journalctl@ceph.mon.a.smithi016.stdout: filter_policy: bloomfilter 2024-09-17T12:27:26.721 INFO:journalctl@ceph.mon.a.smithi016.stdout: whole_key_filtering: 1 2024-09-17T12:27:26.721 INFO:journalctl@ceph.mon.a.smithi016.stdout: verify_compression: 0 2024-09-17T12:27:26.721 INFO:journalctl@ceph.mon.a.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-09-17T12:27:26.721 INFO:journalctl@ceph.mon.a.smithi016.stdout: format_version: 5 2024-09-17T12:27:26.721 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_index_compression: 1 2024-09-17T12:27:26.721 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_align: 0 2024-09-17T12:27:26.721 INFO:journalctl@ceph.mon.a.smithi016.stdout: max_auto_readahead_size: 262144 2024-09-17T12:27:26.722 INFO:journalctl@ceph.mon.a.smithi016.stdout: prepopulate_block_cache: 0 2024-09-17T12:27:26.722 INFO:journalctl@ceph.mon.a.smithi016.stdout: initial_auto_readahead_size: 8192 2024-09-17T12:27:26.722 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T12:27:26.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T12:27:26.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T12:27:26.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compression: NoCompression 2024-09-17T12:27:26.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T12:27:26.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T12:27:26.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T12:27:26.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.num_levels: 7 2024-09-17T12:27:26.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T12:27:26.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T12:27:26.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T12:27:26.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T12:27:26.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T12:27:26.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T12:27:26.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T12:27:26.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T12:27:26.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T12:27:26.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T12:27:26.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T12:27:26.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T12:27:26.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T12:27:26.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T12:27:26.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T12:27:26.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T12:27:26.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T12:27:26.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T12:27:26.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T12:27:26.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T12:27:26.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T12:27:26.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T12:27:26.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T12:27:26.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T12:27:26.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T12:27:26.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T12:27:26.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T12:27:26.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T12:27:26.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T12:27:26.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T12:27:26.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T12:27:26.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T12:27:26.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T12:27:26.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T12:27:26.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T12:27:26.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T12:27:26.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T12:27:26.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T12:27:26.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T12:27:26.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T12:27:26.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T12:27:26.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T12:27:26.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T12:27:26.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T12:27:26.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T12:27:26.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T12:27:26.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T12:27:26.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T12:27:26.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T12:27:26.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T12:27:26.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T12:27:26.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T12:27:26.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T12:27:26.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T12:27:26.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.inplace_update_support: 0 2024-09-17T12:27:26.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T12:27:26.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T12:27:26.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T12:27:26.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T12:27:26.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.bloom_locality: 0 2024-09-17T12:27:26.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.max_successive_merges: 0 2024-09-17T12:27:26.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T12:27:26.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T12:27:26.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T12:27:26.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T12:27:26.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.ttl: 2592000 2024-09-17T12:27:26.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T12:27:26.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T12:27:26.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T12:27:26.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.enable_blob_files: false 2024-09-17T12:27:26.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.min_blob_size: 0 2024-09-17T12:27:26.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T12:27:26.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T12:27:26.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T12:27:26.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T12:27:26.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T12:27:26.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T12:27:26.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T12:27:26.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T12:27:26.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T12:27:26.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12344, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-09-17T12:27:26.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-09-17T12:27:26.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a064515c-6d2f-4b13-9e9e-c1520d5f10d2 2024-09-17T12:27:26.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726576046705347, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-09-17T12:27:26.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-09-17T12:27:27.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726576046713998, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 1938257, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12345, "largest_seqno": 12977, "table_properties": {"data_size": 1934250, "index_size": 2199, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 7126, "raw_average_key_size": 23, "raw_value_size": 1927456, "raw_average_value_size": 6446, "num_data_blocks": 101, "num_entries": 299, "num_filter_entries": 299, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726576046, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a064515c-6d2f-4b13-9e9e-c1520d5f10d2", "db_session_id": "LZI0UXYYEADWCEA4M1XL", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-09-17T12:27:27.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726576046714119, "job": 1, "event": "recovery_finished"} 2024-09-17T12:27:27.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-09-17T12:27:27.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T12:27:27.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T12:27:27.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fe7eaa2e00 2024-09-17T12:27:27.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: DB pointer 0x55fe7eba4000 2024-09-17T12:27:27.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T12:27:27.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T12:27:27.192 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** DB Stats ** 2024-09-17T12:27:27.192 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T12:27:27.192 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T12:27:27.192 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T12:27:27.193 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T12:27:27.193 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T12:27:27.193 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T12:27:27.193 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T12:27:27.193 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-17T12:27:27.193 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-09-17T12:27:27.193 INFO:journalctl@ceph.mon.a.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T12:27:27.193 INFO:journalctl@ceph.mon.a.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T12:27:27.193 INFO:journalctl@ceph.mon.a.smithi016.stdout: L0 1/0 1.85 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 424.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-17T12:27:27.193 INFO:journalctl@ceph.mon.a.smithi016.stdout: L6 1/0 8.38 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-17T12:27:27.193 INFO:journalctl@ceph.mon.a.smithi016.stdout: Sum 2/0 10.23 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 424.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-17T12:27:27.193 INFO:journalctl@ceph.mon.a.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 424.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 424.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T12:27:27.194 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative compaction: 0.00 GB write, 165.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval compaction: 0.00 GB write, 165.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache BinnedLRUCache@0x55fe7ea73610#2 capacity: 512.00 MB usage: 3.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.91 KB,0.000172853%) IndexBlock(1,2.33 KB,0.000444055%) Misc(1,0.00 KB,0%) 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: starting mon.a rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: mon.a@-1(???) e3 preinit fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: mon.a@-1(???).mds e1 new map 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: mon.a@-1(???).mds e1 print_map 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: e1 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T12:27:27.195 INFO:journalctl@ceph.mon.a.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T12:27:27.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: legacy client fscid: -1 2024-09-17T12:27:27.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-17T12:27:27.196 INFO:journalctl@ceph.mon.a.smithi016.stdout: No filesystems configured 2024-09-17T12:27:27.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: mon.a@-1(???).osd e103 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-17T12:27:27.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: mon.a@-1(???).osd e103 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T12:27:27.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: mon.a@-1(???).osd e103 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T12:27:27.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: mon.a@-1(???).osd e103 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T12:27:27.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 ceph-mon[87889]: mon.a@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-09-17T12:27:27.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 bash[87870]: 4a83b787c6059560782c44c0bbc408d98604596747a05d9ab1df11eeb2c274c0 2024-09-17T12:27:27.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:26 smithi016 systemd[1]: Started Ceph mon.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:27:28.526 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: ignoring --setuser ceph since I am not root 2024-09-17T12:27:28.526 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: ignoring --setgroup ceph since I am not root 2024-09-17T12:27:28.527 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mgr[73177]: -- 172.21.15.196:0/2747861373 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b49fb990e0 con 0x55b49fb77400 2024-09-17T12:27:28.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:27:28.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:27:28.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:27:28.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:28.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:28.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:27:28.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 518 B/s rd, 0 op/s 2024-09-17T12:27:28.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:28.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: mon.a calling monitor election 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: monmap epoch 3 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: last_changed 2024-09-17T12:16:57.230811+0000 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: created 2024-09-17T12:15:22.758286+0000 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: min_mon_release 18 (reef) 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: election_strategy: 1 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: 2: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: fsmap 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: mgrmap e35: y(active, since 33s), standbys: x 2024-09-17T12:27:28.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T12:27:28.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T12:27:28.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: daemon iscsi.foo.smithi016.qaboca on smithi016 is in unknown state 2024-09-17T12:27:28.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: from='mgr.15222 ' entity='' 2024-09-17T12:27:28.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:28 smithi196 ceph-mon[27428]: mgrmap e36: y(active, since 33s), standbys: x 2024-09-17T12:27:28.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ignoring --setuser ceph since I am not root 2024-09-17T12:27:28.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ignoring --setgroup ceph since I am not root 2024-09-17T12:27:28.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mgr[79096]: -- 172.21.15.16:0/3948930897 <== mon.2 v2:172.21.15.16:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f371bc30e0 con 0x55f371ba0800 2024-09-17T12:27:28.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:27:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:27:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:27:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:27:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 518 B/s rd, 0 op/s 2024-09-17T12:27:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: mon.a calling monitor election 2024-09-17T12:27:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T12:27:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: monmap epoch 3 2024-09-17T12:27:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:27:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: last_changed 2024-09-17T12:16:57.230811+0000 2024-09-17T12:27:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: created 2024-09-17T12:15:22.758286+0000 2024-09-17T12:27:28.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: min_mon_release 18 (reef) 2024-09-17T12:27:28.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: election_strategy: 1 2024-09-17T12:27:28.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-17T12:27:28.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-17T12:27:28.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: 2: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-09-17T12:27:28.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: fsmap 2024-09-17T12:27:28.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T12:27:28.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: mgrmap e35: y(active, since 33s), standbys: x 2024-09-17T12:27:28.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T12:27:28.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T12:27:28.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: daemon iscsi.foo.smithi016.qaboca on smithi016 is in unknown state 2024-09-17T12:27:28.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: from='mgr.15222 ' entity='' 2024-09-17T12:27:28.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[87889]: mgrmap e36: y(active, since 33s), standbys: x 2024-09-17T12:27:28.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:27:28.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:27:28.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:27:28.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:28.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:28.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: from='mgr.15222 172.21.15.16:0/1212548034' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:27:28.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 518 B/s rd, 0 op/s 2024-09-17T12:27:28.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:28.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: mon.a calling monitor election 2024-09-17T12:27:28.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T12:27:28.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: monmap epoch 3 2024-09-17T12:27:28.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: last_changed 2024-09-17T12:16:57.230811+0000 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: created 2024-09-17T12:15:22.758286+0000 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: min_mon_release 18 (reef) 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: election_strategy: 1 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: 2: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: fsmap 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: mgrmap e35: y(active, since 33s), standbys: x 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: daemon iscsi.foo.smithi016.qaboca on smithi016 is in unknown state 2024-09-17T12:27:28.695 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: from='mgr.15222 ' entity='' 2024-09-17T12:27:28.696 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:28 smithi016 ceph-mon[34212]: mgrmap e36: y(active, since 33s), standbys: x 2024-09-17T12:27:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:33 smithi196 ceph-mon[27428]: Standby manager daemon x restarted 2024-09-17T12:27:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:33 smithi196 ceph-mon[27428]: Standby manager daemon x started 2024-09-17T12:27:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:33 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/3825297242' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:27:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:33 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/3825297242' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:27:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:33 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/3825297242' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:27:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:33 smithi196 ceph-mon[27428]: from='mgr.? 172.21.15.196:0/3825297242' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:27:33.527 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: [17/Sep/2024:12:27:33] ENGINE Bus STARTING 2024-09-17T12:27:33.528 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: CherryPy Checker: 2024-09-17T12:27:33.528 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: The Application mounted at '' has an empty config. 2024-09-17T12:27:33.528 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: 2024-09-17T12:27:33.528 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: [17/Sep/2024:12:27:33] ENGINE Serving on http://:::9283 2024-09-17T12:27:33.528 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:27:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x[73173]: [17/Sep/2024:12:27:33] ENGINE Bus STARTED 2024-09-17T12:27:33.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:33 smithi016 ceph-mon[34212]: Standby manager daemon x restarted 2024-09-17T12:27:33.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:33 smithi016 ceph-mon[34212]: Standby manager daemon x started 2024-09-17T12:27:33.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:33 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/3825297242' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:27:33.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:33 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/3825297242' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:27:33.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:33 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/3825297242' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:27:33.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:33 smithi016 ceph-mon[34212]: from='mgr.? 172.21.15.196:0/3825297242' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:27:33.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:33 smithi016 ceph-mon[87889]: Standby manager daemon x restarted 2024-09-17T12:27:33.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:33 smithi016 ceph-mon[87889]: Standby manager daemon x started 2024-09-17T12:27:33.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:33 smithi016 ceph-mon[87889]: from='mgr.? 172.21.15.196:0/3825297242' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T12:27:33.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:33 smithi016 ceph-mon[87889]: from='mgr.? 172.21.15.196:0/3825297242' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T12:27:33.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:33 smithi016 ceph-mon[87889]: from='mgr.? 172.21.15.196:0/3825297242' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T12:27:33.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:33 smithi016 ceph-mon[87889]: from='mgr.? 172.21.15.196:0/3825297242' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T12:27:34.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: mgrmap e37: y(active, since 38s), standbys: x 2024-09-17T12:27:34.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: Active manager daemon y restarted 2024-09-17T12:27:34.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: Activating manager daemon y 2024-09-17T12:27:34.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T12:27:34.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: mgrmap e38: y(active, starting, since 0.00884591s), standbys: x 2024-09-17T12:27:34.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:27:34.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:27:34.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:27:34.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:27:34.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:27:34.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:27:34.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:27:34.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:27:34.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:27:34.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:27:34.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:27:34.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:27:34.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:27:34.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:27:34.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:27:34.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:27:34.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: Manager daemon y is now available 2024-09-17T12:27:34.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:34.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:27:34.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:34.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:27:34.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-17T12:27:34.431 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:27:34.431 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:34 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-17T12:27:34.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: mgrmap e37: y(active, since 38s), standbys: x 2024-09-17T12:27:34.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: Active manager daemon y restarted 2024-09-17T12:27:34.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: Activating manager daemon y 2024-09-17T12:27:34.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T12:27:34.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: mgrmap e38: y(active, starting, since 0.00884591s), standbys: x 2024-09-17T12:27:34.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:27:34.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:27:34.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:27:34.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: Manager daemon y is now available 2024-09-17T12:27:34.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:34.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:27:34.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:34.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:27:34.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-17T12:27:34.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:27:34.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-17T12:27:34.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: mgrmap e37: y(active, since 38s), standbys: x 2024-09-17T12:27:34.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: Active manager daemon y restarted 2024-09-17T12:27:34.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: Activating manager daemon y 2024-09-17T12:27:34.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T12:27:34.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: mgrmap e38: y(active, starting, since 0.00884591s), standbys: x 2024-09-17T12:27:34.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T12:27:34.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T12:27:34.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T12:27:34.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T12:27:34.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T12:27:34.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T12:27:34.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T12:27:34.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T12:27:34.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T12:27:34.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T12:27:34.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T12:27:34.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T12:27:34.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T12:27:34.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T12:27:34.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T12:27:34.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T12:27:34.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: Manager daemon y is now available 2024-09-17T12:27:34.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:34.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:27:34.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:34.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T12:27:34.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-17T12:27:34.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T12:27:34.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-17T12:27:34.480 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:27:34.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:27:34.457+0000 7f9d37c73640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-17T12:27:34.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:34] ENGINE Bus STARTING 2024-09-17T12:27:34.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: CherryPy Checker: 2024-09-17T12:27:34.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: The Application mounted at '' has an empty config. 2024-09-17T12:27:34.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:27:34.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:34] ENGINE Serving on http://:::9283 2024-09-17T12:27:34.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:34 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:34] ENGINE Bus STARTED 2024-09-17T12:27:35.086 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:27:35.086 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (5m) 35s ago 6m 25.3M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:27:35.086 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (6m) 17s ago 6m 88.5M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:27:35.086 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 unknown 35s ago 4m - - 2024-09-17T12:27:35.086 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (21s) 17s ago 10m 64.3M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:27:35.087 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (53s) 35s ago 11m 492M - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:27:35.087 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (11m) 35s ago 12m 63.1M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 5d21f110d6fc 2024-09-17T12:27:35.087 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (10m) 17s ago 10m 46.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 262ef622777e 2024-09-17T12:27:35.087 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (10m) 35s ago 10m 55.9M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 950b72634364 2024-09-17T12:27:35.087 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (6m) 35s ago 5m 15.8M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:27:35.087 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (5m) 17s ago 5m 15.2M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:27:35.087 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (10m) 35s ago 10m 61.8M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 e30aaffd3b5f 2024-09-17T12:27:35.087 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (9m) 35s ago 9m 54.0M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 3ad94c30c9c9 2024-09-17T12:27:35.087 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (9m) 35s ago 9m 54.9M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 1fa310695876 2024-09-17T12:27:35.087 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (8m) 35s ago 8m 59.6M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 4a4f8eeaff13 2024-09-17T12:27:35.087 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (8m) 17s ago 8m 55.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 afa20056b398 2024-09-17T12:27:35.087 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (8m) 17s ago 8m 50.7M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 f0dcabb8902f 2024-09-17T12:27:35.087 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (7m) 17s ago 7m 59.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:27:35.088 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (7m) 17s ago 7m 61.4M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:27:35.088 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (26s) 17s ago 6m 37.1M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:27:35.088 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (5m) 35s ago 5m 78.2M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:27:35.088 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (5m) 17s ago 5m 79.5M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:27:35.088 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (5m) 35s ago 5m 80.4M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:27:35.088 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (5m) 17s ago 5m 78.4M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:27:35.484 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:27:35.484 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:27:35.484 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2, 2024-09-17T12:27:35.484 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 1 2024-09-17T12:27:35.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:27:35.484 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:27:35.485 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:27:35.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:27:35.485 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:27:35.485 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T12:27:35.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:27:35.485 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:27:35.485 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:27:35.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:27:35.485 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:27:35.485 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 14, 2024-09-17T12:27:35.485 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:27:35.485 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:27:35.485 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:27:35.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:35 smithi196 ceph-mon[27428]: mgrmap e39: y(active, since 1.01154s), standbys: x 2024-09-17T12:27:35.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:35 smithi196 ceph-mon[27428]: from='client.34101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:27:35.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:35 smithi196 ceph-mon[27428]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:27:35.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:35 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:35.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:35 smithi196 ceph-mon[27428]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:27:35.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:35 smithi196 ceph-mon[27428]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:27:35.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-mon[87889]: mgrmap e39: y(active, since 1.01154s), standbys: x 2024-09-17T12:27:35.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-mon[87889]: from='client.34101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:27:35.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-mon[87889]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:27:35.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-mon[87889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:35.808 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-mon[87889]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:27:35.808 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-mon[87889]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:27:35.808 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-mon[34212]: mgrmap e39: y(active, since 1.01154s), standbys: x 2024-09-17T12:27:35.808 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-mon[34212]: from='client.34101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:27:35.808 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-mon[34212]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:27:35.808 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-mon[34212]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:35.808 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-mon[34212]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:27:35.808 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-mon[34212]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:27:35.809 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:27:35.809 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:27:35.809 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:27:35.809 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:27:35.809 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:27:35.809 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:27:35.809 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:27:35.809 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "2/23 daemons upgraded", 2024-09-17T12:27:35.809 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "", 2024-09-17T12:27:35.809 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:27:35.809 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:27:36.116 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:27:35.813+0000 7f9cf5767640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:12:27:35] ENGINE Error in HTTPServer.serve 2024-09-17T12:27:36.116 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: Traceback (most recent call last): 2024-09-17T12:27:36.116 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:27:36.116 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: self._connections.run(self.expiration_interval) 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: self._run(expiration_interval) 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: s = self.context.wrap_socket( 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: return self.sslsocket_class._create( 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: self.do_handshake() 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:27:36.117 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: self._sslobj.do_handshake() 2024-09-17T12:27:36.118 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:27:36.118 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:27:36.173 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-09-17T12:27:36.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:27:36] "GET /metrics HTTP/1.1" 200 35704 "" "Prometheus/2.43.0" 2024-09-17T12:27:36.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:36 smithi196 ceph-mon[27428]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T12:27:36.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:36 smithi196 ceph-mon[27428]: Cluster is now healthy 2024-09-17T12:27:36.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:36 smithi196 ceph-mon[27428]: mgrmap e40: y(active, since 2s), standbys: x 2024-09-17T12:27:36.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:36 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/2812304652' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:27:36.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:36 smithi196 ceph-mon[27428]: [17/Sep/2024:12:27:35] ENGINE Bus STARTING 2024-09-17T12:27:36.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:36 smithi196 ceph-mon[27428]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:27:36.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:36 smithi196 ceph-mon[27428]: [17/Sep/2024:12:27:35] ENGINE Serving on https://172.21.15.16:7150 2024-09-17T12:27:36.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:36 smithi196 ceph-mon[27428]: [17/Sep/2024:12:27:35] ENGINE Error in HTTPServer.serve 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._run(expiration_interval) 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: s = self.context.wrap_socket( 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.sslsocket_class._create( 2024-09-17T12:27:36.778 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:27:36.779 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.do_handshake() 2024-09-17T12:27:36.779 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:27:36.779 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._sslobj.do_handshake() 2024-09-17T12:27:36.779 INFO:journalctl@ceph.mon.b.smithi196.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:27:36.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:36 smithi196 ceph-mon[27428]: [17/Sep/2024:12:27:35] ENGINE Serving on http://172.21.15.16:8765 2024-09-17T12:27:36.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:36 smithi196 ceph-mon[27428]: [17/Sep/2024:12:27:35] ENGINE Bus STARTED 2024-09-17T12:27:36.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:36 smithi196 ceph-mon[27428]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:36.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:36 smithi196 ceph-mon[27428]: from='client.? 172.21.15.16:0/3120911006' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:27:36.843 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[34212]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T12:27:36.843 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[34212]: Cluster is now healthy 2024-09-17T12:27:36.843 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[34212]: mgrmap e40: y(active, since 2s), standbys: x 2024-09-17T12:27:36.843 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/2812304652' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:27:36.843 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[34212]: [17/Sep/2024:12:27:35] ENGINE Bus STARTING 2024-09-17T12:27:36.843 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[34212]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:27:36.843 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[34212]: [17/Sep/2024:12:27:35] ENGINE Serving on https://172.21.15.16:7150 2024-09-17T12:27:36.843 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[34212]: [17/Sep/2024:12:27:35] ENGINE Error in HTTPServer.serve 2024-09-17T12:27:36.843 INFO:journalctl@ceph.mon.c.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:27:36.843 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:27:36.843 INFO:journalctl@ceph.mon.c.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:27:36.843 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:27:36.844 INFO:journalctl@ceph.mon.c.smithi016.stdout: self._run(expiration_interval) 2024-09-17T12:27:36.844 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:27:36.844 INFO:journalctl@ceph.mon.c.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:27:36.844 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:27:36.844 INFO:journalctl@ceph.mon.c.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:27:36.844 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:27:36.844 INFO:journalctl@ceph.mon.c.smithi016.stdout: s = self.context.wrap_socket( 2024-09-17T12:27:36.844 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:27:36.844 INFO:journalctl@ceph.mon.c.smithi016.stdout: return self.sslsocket_class._create( 2024-09-17T12:27:36.844 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:27:36.844 INFO:journalctl@ceph.mon.c.smithi016.stdout: self.do_handshake() 2024-09-17T12:27:36.844 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:27:36.844 INFO:journalctl@ceph.mon.c.smithi016.stdout: self._sslobj.do_handshake() 2024-09-17T12:27:36.845 INFO:journalctl@ceph.mon.c.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:27:36.845 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[34212]: [17/Sep/2024:12:27:35] ENGINE Serving on http://172.21.15.16:8765 2024-09-17T12:27:36.845 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[34212]: [17/Sep/2024:12:27:35] ENGINE Bus STARTED 2024-09-17T12:27:36.845 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[34212]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:36.845 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[34212]: from='client.? 172.21.15.16:0/3120911006' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:27:36.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[87889]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T12:27:36.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[87889]: Cluster is now healthy 2024-09-17T12:27:36.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[87889]: mgrmap e40: y(active, since 2s), standbys: x 2024-09-17T12:27:36.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/2812304652' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:27:36.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[87889]: [17/Sep/2024:12:27:35] ENGINE Bus STARTING 2024-09-17T12:27:36.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[87889]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[87889]: [17/Sep/2024:12:27:35] ENGINE Serving on https://172.21.15.16:7150 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[87889]: [17/Sep/2024:12:27:35] ENGINE Error in HTTPServer.serve 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-09-17T12:27:36.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T12:27:36.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-09-17T12:27:36.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T12:27:36.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-09-17T12:27:36.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T12:27:36.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-09-17T12:27:36.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T12:27:36.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[87889]: [17/Sep/2024:12:27:35] ENGINE Serving on http://172.21.15.16:8765 2024-09-17T12:27:36.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[87889]: [17/Sep/2024:12:27:35] ENGINE Bus STARTED 2024-09-17T12:27:36.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:36.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:36 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/3120911006' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T12:27:38.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:38 smithi196 ceph-mon[27428]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:27:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:38 smithi196 ceph-mon[27428]: mgrmap e41: y(active, since 4s), standbys: x 2024-09-17T12:27:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:38 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:38.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:38 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:38.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:38 smithi016 ceph-mon[87889]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:27:38.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:38 smithi016 ceph-mon[87889]: mgrmap e41: y(active, since 4s), standbys: x 2024-09-17T12:27:38.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:38 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:38.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:38 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:38.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:38 smithi016 ceph-mon[34212]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:27:38.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:38 smithi016 ceph-mon[34212]: mgrmap e41: y(active, since 4s), standbys: x 2024-09-17T12:27:38.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:38 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:38.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:38 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.705 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:39 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.705 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:39 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.705 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:39 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:39 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:39 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:39 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:39 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:39 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:39 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:39 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:39 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:39 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:39 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:39 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:39 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:39 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:39 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:39.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:39 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:41.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:41 smithi196 ceph-mon[27428]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:27:41.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:41 smithi016 ceph-mon[87889]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:27:41.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:41 smithi016 ceph-mon[34212]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:27:43.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T12:27:43.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:43.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:27:43.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:27:43.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:27:43.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:27:43.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:27:43.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:43.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:27:43.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T12:27:43.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T12:27:43.225 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T12:27:43.225 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.226 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.226 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:43.226 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:27:43.226 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:27:43.226 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:27:43.226 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:27:43.226 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:27:43.226 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.226 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.226 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.226 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.226 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.226 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:43.227 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:27:43.227 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T12:27:43.227 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:42 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T12:27:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T12:27:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:27:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-17T12:27:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-17T12:27:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:27:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T12:27:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:43.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:43.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:27:43.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T12:27:43.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:42 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T12:27:44.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[87889]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:27:44.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[87889]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:27:44.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[87889]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:27:44.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[87889]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:27:44.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[87889]: Upgrade: It appears safe to stop mon.c 2024-09-17T12:27:44.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:44.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:27:44.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:27:44.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:44.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[34212]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:27:44.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[34212]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:27:44.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[34212]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:27:44.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[34212]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:27:44.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[34212]: Upgrade: It appears safe to stop mon.c 2024-09-17T12:27:44.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[34212]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:44.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:27:44.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:27:44.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:43 smithi016 ceph-mon[34212]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:44.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:43 smithi196 ceph-mon[27428]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:27:44.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:43 smithi196 ceph-mon[27428]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-17T12:27:44.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:43 smithi196 ceph-mon[27428]: Updating smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:27:44.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:43 smithi196 ceph-mon[27428]: Updating smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T12:27:44.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:43 smithi196 ceph-mon[27428]: Upgrade: It appears safe to stop mon.c 2024-09-17T12:27:44.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:43 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:44.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:43 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:27:44.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:43 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:27:44.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:43 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:45.268 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:44 smithi016 ceph-mon[87889]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-17T12:27:45.268 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:44 smithi016 ceph-mon[87889]: Upgrade: Updating mon.c 2024-09-17T12:27:45.268 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:44 smithi016 ceph-mon[87889]: Deploying daemon mon.c on smithi016 2024-09-17T12:27:45.268 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:44 smithi016 ceph-mon[34212]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-17T12:27:45.268 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:44 smithi016 ceph-mon[34212]: Upgrade: Updating mon.c 2024-09-17T12:27:45.268 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:44 smithi016 ceph-mon[34212]: Deploying daemon mon.c on smithi016 2024-09-17T12:27:45.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:44 smithi196 ceph-mon[27428]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-17T12:27:45.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:44 smithi196 ceph-mon[27428]: Upgrade: Updating mon.c 2024-09-17T12:27:45.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:44 smithi196 ceph-mon[27428]: Deploying daemon mon.c on smithi016 2024-09-17T12:27:45.836 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:45 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:45] ENGINE Bus STOPPING 2024-09-17T12:27:45.836 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:45 smithi016 systemd[1]: Stopping Ceph mon.c for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:27:45.836 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:45 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c[34207]: 2024-09-17T12:27:45.803+0000 7f3eac7e7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:27:45.836 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:45 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c[34207]: 2024-09-17T12:27:45.803+0000 7f3eac7e7640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-17T12:27:46.146 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:45 smithi016 podman[89622]: 2024-09-17 12:27:45.834555114 +0000 UTC m=+0.110610296 container died 950b726343643eb0826912ae44363bc341a79460cadbcd27937cd85d9a55afe5 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-17T12:27:46.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:46 smithi016 podman[89622]: 2024-09-17 12:27:46.145384776 +0000 UTC m=+0.421439958 container cleanup 950b726343643eb0826912ae44363bc341a79460cadbcd27937cd85d9a55afe5 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-17T12:27:46.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:46 smithi016 bash[89622]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c 2024-09-17T12:27:46.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:46 smithi016 podman[89637]: 2024-09-17 12:27:46.330354829 +0000 UTC m=+0.498203147 container remove 950b726343643eb0826912ae44363bc341a79460cadbcd27937cd85d9a55afe5 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:27:46.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:27:46] "GET /metrics HTTP/1.1" 200 35704 "" "Prometheus/2.43.0" 2024-09-17T12:27:46.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T12:27:46.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:46] ENGINE Bus STOPPED 2024-09-17T12:27:46.905 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:46 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-17T12:27:46.905 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:46 smithi016 systemd[1]: Stopped Ceph mon.c for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:27:46.905 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:46 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.c.service: Consumed 8.095s CPU time. 2024-09-17T12:27:46.905 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:46 smithi016 systemd[1]: Starting Ceph mon.c for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:27:47.185 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:46 smithi016 podman[89765]: 2024-09-17 12:27:46.807024698 +0000 UTC m=+0.016534856 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:27:47.185 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:46 smithi016 podman[89765]: 2024-09-17 12:27:46.965264789 +0000 UTC m=+0.174774927 container create aa5074acd22aae41d6cb06c18d239991acc17c796d6eda72a6813879f7af43f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-7498493, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:27:47.185 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 podman[89765]: 2024-09-17 12:27:47.184150189 +0000 UTC m=+0.393660347 container init aa5074acd22aae41d6cb06c18d239991acc17c796d6eda72a6813879f7af43f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-7498493, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True) 2024-09-17T12:27:47.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:47] ENGINE Bus STARTING 2024-09-17T12:27:47.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 podman[89765]: 2024-09-17 12:27:47.187606701 +0000 UTC m=+0.397116847 container start aa5074acd22aae41d6cb06c18d239991acc17c796d6eda72a6813879f7af43f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=main-7498493, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:27:47.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T12:27:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev), process ceph-mon, pid 2 2024-09-17T12:27:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: pidfile_write: ignore empty --pid-file 2024-09-17T12:27:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: load: jerasure load: lrc 2024-09-17T12:27:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: RocksDB version: 7.9.2 2024-09-17T12:27:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Git sha 0 2024-09-17T12:27:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Compile date 2024-09-15 20:52:36 2024-09-17T12:27:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: DB SUMMARY 2024-09-17T12:27:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: DB Session ID: ZW0LICL4BTY5C9YYBYT7 2024-09-17T12:27:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: CURRENT file: CURRENT 2024-09-17T12:27:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T12:27:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1390 Bytes 2024-09-17T12:27:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-09-17T12:27:47.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 6370592 ; 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.error_if_exists: 0 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.create_if_missing: 0 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.paranoid_checks: 1 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.env: 0x559ddafff120 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.info_log: 0x559ddcae7700 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.statistics: (nil) 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.use_fsync: 0 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_log_file_size: 0 2024-09-17T12:27:47.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T12:27:47.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T12:27:47.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T12:27:47.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T12:27:47.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.allow_fallocate: 1 2024-09-17T12:27:47.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T12:27:47.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T12:27:47.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.use_direct_reads: 0 2024-09-17T12:27:47.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T12:27:47.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T12:27:47.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.db_log_dir: 2024-09-17T12:27:47.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.wal_dir: 2024-09-17T12:27:47.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T12:27:47.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T12:27:47.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T12:27:47.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T12:27:47.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T12:27:47.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T12:27:47.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T12:27:47.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T12:27:47.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.write_buffer_manager: 0x559ddcaea1e0 2024-09-17T12:27:47.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T12:27:47.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T12:27:47.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T12:27:47.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T12:27:47.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T12:27:47.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T12:27:47.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T12:27:47.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T12:27:47.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.unordered_write: 0 2024-09-17T12:27:47.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T12:27:47.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T12:27:47.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T12:27:47.447 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T12:27:47.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.row_cache: None 2024-09-17T12:27:47.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.wal_filter: None 2024-09-17T12:27:47.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T12:27:47.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T12:27:47.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.two_write_queues: 0 2024-09-17T12:27:47.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T12:27:47.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.wal_compression: 0 2024-09-17T12:27:47.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.atomic_flush: 0 2024-09-17T12:27:47.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T12:27:47.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T12:27:47.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T12:27:47.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.log_readahead_size: 0 2024-09-17T12:27:47.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T12:27:47.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T12:27:47.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T12:27:47.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T12:27:47.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T12:27:47.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T12:27:47.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T12:27:47.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_background_jobs: 2 2024-09-17T12:27:47.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_background_compactions: -1 2024-09-17T12:27:47.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_subcompactions: 1 2024-09-17T12:27:47.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T12:27:47.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T12:27:47.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T12:27:47.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T12:27:47.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T12:27:47.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T12:27:47.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T12:27:47.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T12:27:47.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_open_files: -1 2024-09-17T12:27:47.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T12:27:47.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T12:27:47.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T12:27:47.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T12:27:47.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_background_flushes: -1 2024-09-17T12:27:47.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Compression algorithms supported: 2024-09-17T12:27:47.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: kZSTD supported: 0 2024-09-17T12:27:47.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: kXpressCompression supported: 0 2024-09-17T12:27:47.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: kBZip2Compression supported: 0 2024-09-17T12:27:47.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T12:27:47.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: kLZ4Compression supported: 1 2024-09-17T12:27:47.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: kZlibCompression supported: 1 2024-09-17T12:27:47.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T12:27:47.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: kSnappyCompression supported: 1 2024-09-17T12:27:47.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T12:27:47.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T12:27:47.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 2024-09-17T12:27:47.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T12:27:47.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T12:27:47.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.merge_operator: 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compaction_filter: None 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559ddcae7320) 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_type: 0 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout: data_block_index_type: 0 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_shortening: 1 2024-09-17T12:27:47.452 INFO:journalctl@ceph.mon.c.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T12:27:47.453 INFO:journalctl@ceph.mon.c.smithi016.stdout: checksum: 4 2024-09-17T12:27:47.453 INFO:journalctl@ceph.mon.c.smithi016.stdout: no_block_cache: 0 2024-09-17T12:27:47.453 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache: 0x559ddcadb610 2024-09-17T12:27:47.453 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-09-17T12:27:47.453 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_options: 2024-09-17T12:27:47.453 INFO:journalctl@ceph.mon.c.smithi016.stdout: capacity : 536870912 2024-09-17T12:27:47.453 INFO:journalctl@ceph.mon.c.smithi016.stdout: num_shard_bits : 4 2024-09-17T12:27:47.453 INFO:journalctl@ceph.mon.c.smithi016.stdout: strict_capacity_limit : 0 2024-09-17T12:27:47.453 INFO:journalctl@ceph.mon.c.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-09-17T12:27:47.453 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_compressed: (nil) 2024-09-17T12:27:47.453 INFO:journalctl@ceph.mon.c.smithi016.stdout: persistent_cache: (nil) 2024-09-17T12:27:47.453 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_size: 4096 2024-09-17T12:27:47.453 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_size_deviation: 10 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_restart_interval: 16 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_block_restart_interval: 1 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: metadata_block_size: 4096 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: partition_filters: 0 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: use_delta_encoding: 1 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: filter_policy: bloomfilter 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: whole_key_filtering: 1 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: verify_compression: 0 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: format_version: 5 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: enable_index_compression: 1 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_align: 0 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: max_auto_readahead_size: 262144 2024-09-17T12:27:47.454 INFO:journalctl@ceph.mon.c.smithi016.stdout: prepopulate_block_cache: 0 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout: initial_auto_readahead_size: 8192 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compression: NoCompression 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.num_levels: 7 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T12:27:47.455 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T12:27:47.456 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T12:27:47.457 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T12:27:47.458 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.inplace_update_support: 0 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.bloom_locality: 0 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.max_successive_merges: 0 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T12:27:47.459 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.ttl: 2592000 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.enable_blob_files: false 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.min_blob_size: 0 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T12:27:47.460 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T12:27:47.461 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T12:27:47.461 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T12:27:47.461 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 12962, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-09-17T12:27:47.461 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-17T12:27:47.461 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: beb14d0f-605c-4ec2-818d-c14fa1c35b70 2024-09-17T12:27:47.461 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726576067215523, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-17T12:27:47.461 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-17T12:27:47.461 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726576067229744, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3895832, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12967, "largest_seqno": 14131, "table_properties": {"data_size": 3889744, "index_size": 3699, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1349, "raw_key_size": 12479, "raw_average_key_size": 23, "raw_value_size": 3878259, "raw_average_value_size": 7415, "num_data_blocks": 168, "num_entries": 523, "num_filter_entries": 523, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726576067, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "beb14d0f-605c-4ec2-818d-c14fa1c35b70", "db_session_id": "ZW0LICL4BTY5C9YYBYT7", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-17T12:27:47.461 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726576067229882, "job": 1, "event": "recovery_finished"} 2024-09-17T12:27:47.461 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-17T12:27:47.461 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T12:27:47.461 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T12:27:47.461 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559ddcb0ae00 2024-09-17T12:27:47.462 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: DB pointer 0x559ddcc0c000 2024-09-17T12:27:47.462 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T12:27:47.462 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T12:27:47.462 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** DB Stats ** 2024-09-17T12:27:47.462 INFO:journalctl@ceph.mon.c.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T12:27:47.462 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T12:27:47.462 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T12:27:47.462 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T12:27:47.462 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T12:27:47.462 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T12:27:47.462 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T12:27:47.462 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-09-17T12:27:47.462 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** Compaction Stats [default] ** 2024-09-17T12:27:47.463 INFO:journalctl@ceph.mon.c.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T12:27:47.463 INFO:journalctl@ceph.mon.c.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T12:27:47.463 INFO:journalctl@ceph.mon.c.smithi016.stdout: L0 1/0 3.72 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 627.5 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-09-17T12:27:47.463 INFO:journalctl@ceph.mon.c.smithi016.stdout: L6 1/0 8.38 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-17T12:27:47.463 INFO:journalctl@ceph.mon.c.smithi016.stdout: Sum 2/0 12.10 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 627.5 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-09-17T12:27:47.463 INFO:journalctl@ceph.mon.c.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 627.5 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-09-17T12:27:47.463 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-09-17T12:27:47.463 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** Compaction Stats [default] ** 2024-09-17T12:27:47.463 INFO:journalctl@ceph.mon.c.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T12:27:47.463 INFO:journalctl@ceph.mon.c.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T12:27:47.463 INFO:journalctl@ceph.mon.c.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 627.5 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-09-17T12:27:47.463 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-09-17T12:27:47.463 INFO:journalctl@ceph.mon.c.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative compaction: 0.00 GB write, 218.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval compaction: 0.00 GB write, 218.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: Block cache BinnedLRUCache@0x559ddcadb610#2 capacity: 512.00 MB usage: 5.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.41 KB,0.000268221%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-09-17T12:27:47.464 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: starting mon.c rank 2 at public addrs [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] at bind addrs [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: mon.c@-1(???) e3 preinit fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: mon.c@-1(???).mds e1 new map 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: mon.c@-1(???).mds e1 print_map 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout: e1 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout: legacy client fscid: -1 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout: No filesystems configured 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: mon.c@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: mon.c@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: mon.c@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T12:27:47.465 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: mon.c@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T12:27:47.466 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-mon[89784]: mon.c@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-09-17T12:27:47.466 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 bash[89765]: aa5074acd22aae41d6cb06c18d239991acc17c796d6eda72a6813879f7af43f6 2024-09-17T12:27:47.466 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:47 smithi016 systemd[1]: Started Ceph mon.c for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:27:47.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:47] ENGINE Serving on http://:::9283 2024-09-17T12:27:47.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:47] ENGINE Bus STARTED 2024-09-17T12:27:48.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:48.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:48.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:48.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: mon.c calling monitor election 2024-09-17T12:27:48.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: mon.a calling monitor election 2024-09-17T12:27:48.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T12:27:48.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: monmap epoch 3 2024-09-17T12:27:48.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:27:48.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: last_changed 2024-09-17T12:16:57.230811+0000 2024-09-17T12:27:48.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: created 2024-09-17T12:15:22.758286+0000 2024-09-17T12:27:48.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: min_mon_release 18 (reef) 2024-09-17T12:27:48.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: election_strategy: 1 2024-09-17T12:27:48.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-17T12:27:48.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-17T12:27:48.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: 2: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-09-17T12:27:48.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: fsmap 2024-09-17T12:27:48.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T12:27:48.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: mgrmap e41: y(active, since 13s), standbys: x 2024-09-17T12:27:48.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: overall HEALTH_OK 2024-09-17T12:27:48.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:48.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:48.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:48.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[87889]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:48.689 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:48.689 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:48.689 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:48.689 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: mon.c calling monitor election 2024-09-17T12:27:48.689 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: mon.a calling monitor election 2024-09-17T12:27:48.689 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T12:27:48.689 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: monmap epoch 3 2024-09-17T12:27:48.689 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:27:48.689 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: last_changed 2024-09-17T12:16:57.230811+0000 2024-09-17T12:27:48.689 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: created 2024-09-17T12:15:22.758286+0000 2024-09-17T12:27:48.689 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: min_mon_release 18 (reef) 2024-09-17T12:27:48.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: election_strategy: 1 2024-09-17T12:27:48.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-17T12:27:48.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-17T12:27:48.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: 2: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-09-17T12:27:48.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: fsmap 2024-09-17T12:27:48.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T12:27:48.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: mgrmap e41: y(active, since 13s), standbys: x 2024-09-17T12:27:48.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: overall HEALTH_OK 2024-09-17T12:27:48.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:48.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:48.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:48.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:48 smithi016 ceph-mon[89784]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:48.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:48.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:48.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:27:48.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: mon.c calling monitor election 2024-09-17T12:27:48.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: mon.a calling monitor election 2024-09-17T12:27:48.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T12:27:48.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: monmap epoch 3 2024-09-17T12:27:48.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:27:48.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: last_changed 2024-09-17T12:16:57.230811+0000 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: created 2024-09-17T12:15:22.758286+0000 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: min_mon_release 18 (reef) 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: election_strategy: 1 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: 2: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: fsmap 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: mgrmap e41: y(active, since 13s), standbys: x 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: overall HEALTH_OK 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:48.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:48 smithi196 ceph-mon[27428]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:50.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:49 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:50.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:27:50.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:49 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:50.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:27:50.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:49 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:50.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:49 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T12:27:51.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:50 smithi016 ceph-mon[87889]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:51.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:50 smithi016 ceph-mon[89784]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:51.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:50 smithi196 ceph-mon[27428]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-17T12:27:52.528 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:52 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:27:52.240+0000 7f9d37c73640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-17T12:27:52.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:52 smithi016 ceph-mon[87889]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-17T12:27:52.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:52 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:52.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:52 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:52.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:52 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:52.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:52 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:52.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:52 smithi016 ceph-mon[89784]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-17T12:27:52.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:52 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:52.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:52 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:52.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:52 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:52.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:52 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:53.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:52 smithi196 ceph-mon[27428]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-17T12:27:53.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:52 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:53.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:52 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:53.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:52 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:53.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:52 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:54.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:54 smithi016 ceph-mon[87889]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:27:54.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:54 smithi016 ceph-mon[89784]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:27:55.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:54 smithi196 ceph-mon[27428]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:27:56.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:27:56] "GET /metrics HTTP/1.1" 200 38436 "" "Prometheus/2.43.0" 2024-09-17T12:27:56.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:27:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:27:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T12:27:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T12:27:56.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[87889]: Upgrade: It appears safe to stop mon.b 2024-09-17T12:27:56.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:56.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:56.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:56.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:27:56.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:56.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:56.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:27:56.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T12:27:56.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T12:27:56.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:56 smithi016 ceph-mon[89784]: Upgrade: It appears safe to stop mon.b 2024-09-17T12:27:56.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:56 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:56.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:56 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:56.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:56 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:56.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:56 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T12:27:56.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:56 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:56.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:56 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T12:27:56.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:56 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T12:27:56.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:56 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T12:27:56.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:56 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T12:27:56.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:56 smithi196 ceph-mon[27428]: Upgrade: It appears safe to stop mon.b 2024-09-17T12:27:57.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:57 smithi196 ceph-mon[27428]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:27:57.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:57 smithi196 ceph-mon[27428]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:57.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:57 smithi196 ceph-mon[27428]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:57.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:57 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:27:57.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:57 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:27:57.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:57 smithi196 ceph-mon[27428]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:57 smithi016 ceph-mon[87889]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:27:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:57 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:57 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:57 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:27:57.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:57 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:27:57.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:57 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:57 smithi016 ceph-mon[89784]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:27:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:57 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:27:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:57 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:27:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:57 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T12:27:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:57 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T12:27:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:57 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T12:27:58.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:58 smithi196 systemd[1]: Stopping Ceph mon.b for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:27:58.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:58 smithi196 ceph-mon[27428]: Upgrade: Updating mon.b 2024-09-17T12:27:58.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:58 smithi196 ceph-mon[27428]: Deploying daemon mon.b on smithi196 2024-09-17T12:27:58.402 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:58 smithi016 ceph-mon[87889]: Upgrade: Updating mon.b 2024-09-17T12:27:58.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:27:58 smithi016 ceph-mon[87889]: Deploying daemon mon.b on smithi196 2024-09-17T12:27:58.403 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:58 smithi016 ceph-mon[89784]: Upgrade: Updating mon.b 2024-09-17T12:27:58.403 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:27:58 smithi016 ceph-mon[89784]: Deploying daemon mon.b on smithi196 2024-09-17T12:27:58.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:58 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:58] ENGINE Bus STOPPING 2024-09-17T12:27:58.707 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b[27412]: 2024-09-17T12:27:58.379+0000 7fe2a5a63640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:27:58.707 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:58 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b[27412]: 2024-09-17T12:27:58.379+0000 7fe2a5a63640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-17T12:27:58.707 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:58 smithi196 podman[80179]: 2024-09-17 12:27:58.400911407 +0000 UTC m=+0.141478325 container died 262ef622777e9a56bb7eda009090d057f7f541d41b5dd406cfc1f48fe8ee21f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-17T12:27:59.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:58 smithi196 podman[80179]: 2024-09-17 12:27:58.822241033 +0000 UTC m=+0.562807950 container cleanup 262ef622777e9a56bb7eda009090d057f7f541d41b5dd406cfc1f48fe8ee21f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b, GIT_CLEAN=True, ceph=True, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-17T12:27:59.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:58 smithi196 bash[80179]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b 2024-09-17T12:27:59.157 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:58 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T12:27:59.157 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:58 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:58] ENGINE Bus STOPPED 2024-09-17T12:27:59.157 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:58 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:58] ENGINE Bus STARTING 2024-09-17T12:27:59.157 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:58 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:58] ENGINE Serving on http://:::9283 2024-09-17T12:27:59.157 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:27:58 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:27:58] ENGINE Bus STARTED 2024-09-17T12:27:59.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 podman[80191]: 2024-09-17 12:27:59.039235347 +0000 UTC m=+0.636755204 container remove 262ef622777e9a56bb7eda009090d057f7f541d41b5dd406cfc1f48fe8ee21f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-17T12:27:59.362 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.b.service: Deactivated successfully. 2024-09-17T12:27:59.362 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 systemd[1]: Stopped Ceph mon.b for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:27:59.362 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.b.service: Consumed 8.504s CPU time. 2024-09-17T12:27:59.681 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 systemd[1]: Starting Ceph mon.b for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:27:59.682 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 podman[80320]: 2024-09-17 12:27:59.476615324 +0000 UTC m=+0.016776516 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:27:59.682 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 podman[80320]: 2024-09-17 12:27:59.582306376 +0000 UTC m=+0.122467550 container create dc48dfc9e40cb61472b185c889f77662327130265661fdd9ef1d75eaf04ceb5a (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=main-7498493) 2024-09-17T12:28:00.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 podman[80320]: 2024-09-17 12:27:59.801889236 +0000 UTC m=+0.342050423 container init dc48dfc9e40cb61472b185c889f77662327130265661fdd9ef1d75eaf04ceb5a (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-7498493) 2024-09-17T12:28:00.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 podman[80320]: 2024-09-17 12:27:59.805293341 +0000 UTC m=+0.345454516 container start dc48dfc9e40cb61472b185c889f77662327130265661fdd9ef1d75eaf04ceb5a (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b, RELEASE=main-7498493, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=) 2024-09-17T12:28:00.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T12:28:00.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev), process ceph-mon, pid 2 2024-09-17T12:28:00.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: pidfile_write: ignore empty --pid-file 2024-09-17T12:28:00.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: load: jerasure load: lrc 2024-09-17T12:28:00.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: RocksDB version: 7.9.2 2024-09-17T12:28:00.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Git sha 0 2024-09-17T12:28:00.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Compile date 2024-09-15 20:52:36 2024-09-17T12:28:00.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: DB SUMMARY 2024-09-17T12:28:00.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: DB Session ID: PXREHE832XXUXVWGIZEV 2024-09-17T12:28:00.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: CURRENT file: CURRENT 2024-09-17T12:28:00.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T12:28:00.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1390 Bytes 2024-09-17T12:28:00.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-09-17T12:28:00.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 7068307 ; 2024-09-17T12:28:00.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.error_if_exists: 0 2024-09-17T12:28:00.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.create_if_missing: 0 2024-09-17T12:28:00.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.paranoid_checks: 1 2024-09-17T12:28:00.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T12:28:00.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T12:28:00.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T12:28:00.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.env: 0x55841f171120 2024-09-17T12:28:00.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T12:28:00.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.info_log: 0x5584207df700 2024-09-17T12:28:00.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T12:28:00.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.statistics: (nil) 2024-09-17T12:28:00.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.use_fsync: 0 2024-09-17T12:28:00.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_log_file_size: 0 2024-09-17T12:28:00.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T12:28:00.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T12:28:00.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T12:28:00.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T12:28:00.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.allow_fallocate: 1 2024-09-17T12:28:00.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T12:28:00.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T12:28:00.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.use_direct_reads: 0 2024-09-17T12:28:00.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T12:28:00.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T12:28:00.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.db_log_dir: 2024-09-17T12:28:00.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.wal_dir: 2024-09-17T12:28:00.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T12:28:00.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T12:28:00.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T12:28:00.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T12:28:00.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T12:28:00.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T12:28:00.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T12:28:00.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T12:28:00.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.write_buffer_manager: 0x5584207e21e0 2024-09-17T12:28:00.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T12:28:00.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T12:28:00.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T12:28:00.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T12:28:00.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T12:28:00.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T12:28:00.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T12:28:00.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T12:28:00.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.unordered_write: 0 2024-09-17T12:28:00.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T12:28:00.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T12:28:00.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T12:28:00.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T12:28:00.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.row_cache: None 2024-09-17T12:28:00.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.wal_filter: None 2024-09-17T12:28:00.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T12:28:00.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T12:28:00.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.two_write_queues: 0 2024-09-17T12:28:00.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T12:28:00.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.wal_compression: 0 2024-09-17T12:28:00.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.atomic_flush: 0 2024-09-17T12:28:00.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T12:28:00.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T12:28:00.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T12:28:00.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.log_readahead_size: 0 2024-09-17T12:28:00.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T12:28:00.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T12:28:00.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T12:28:00.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T12:28:00.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T12:28:00.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T12:28:00.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T12:28:00.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_background_jobs: 2 2024-09-17T12:28:00.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_background_compactions: -1 2024-09-17T12:28:00.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_subcompactions: 1 2024-09-17T12:28:00.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T12:28:00.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T12:28:00.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T12:28:00.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T12:28:00.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T12:28:00.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T12:28:00.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T12:28:00.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T12:28:00.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_open_files: -1 2024-09-17T12:28:00.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T12:28:00.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T12:28:00.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T12:28:00.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T12:28:00.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_background_flushes: -1 2024-09-17T12:28:00.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Compression algorithms supported: 2024-09-17T12:28:00.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: kZSTD supported: 0 2024-09-17T12:28:00.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: kXpressCompression supported: 0 2024-09-17T12:28:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: kBZip2Compression supported: 0 2024-09-17T12:28:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T12:28:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: kLZ4Compression supported: 1 2024-09-17T12:28:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: kZlibCompression supported: 1 2024-09-17T12:28:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T12:28:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: kSnappyCompression supported: 1 2024-09-17T12:28:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T12:28:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T12:28:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 2024-09-17T12:28:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T12:28:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T12:28:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.merge_operator: 2024-09-17T12:28:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compaction_filter: None 2024-09-17T12:28:00.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T12:28:00.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T12:28:00.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T12:28:00.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T12:28:00.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5584207df320) 2024-09-17T12:28:00.290 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks: 1 2024-09-17T12:28:00.290 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T12:28:00.290 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T12:28:00.290 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_top_level_index_and_filter: 1 2024-09-17T12:28:00.291 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_type: 0 2024-09-17T12:28:00.291 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_index_type: 0 2024-09-17T12:28:00.291 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_shortening: 1 2024-09-17T12:28:00.291 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T12:28:00.291 INFO:journalctl@ceph.mon.b.smithi196.stdout: checksum: 4 2024-09-17T12:28:00.291 INFO:journalctl@ceph.mon.b.smithi196.stdout: no_block_cache: 0 2024-09-17T12:28:00.291 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache: 0x5584207d3610 2024-09-17T12:28:00.291 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_name: BinnedLRUCache 2024-09-17T12:28:00.291 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_options: 2024-09-17T12:28:00.291 INFO:journalctl@ceph.mon.b.smithi196.stdout: capacity : 536870912 2024-09-17T12:28:00.291 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_shard_bits : 4 2024-09-17T12:28:00.291 INFO:journalctl@ceph.mon.b.smithi196.stdout: strict_capacity_limit : 0 2024-09-17T12:28:00.292 INFO:journalctl@ceph.mon.b.smithi196.stdout: high_pri_pool_ratio: 0.000 2024-09-17T12:28:00.292 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_compressed: (nil) 2024-09-17T12:28:00.292 INFO:journalctl@ceph.mon.b.smithi196.stdout: persistent_cache: (nil) 2024-09-17T12:28:00.292 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size: 4096 2024-09-17T12:28:00.292 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size_deviation: 10 2024-09-17T12:28:00.292 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_restart_interval: 16 2024-09-17T12:28:00.292 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_block_restart_interval: 1 2024-09-17T12:28:00.292 INFO:journalctl@ceph.mon.b.smithi196.stdout: metadata_block_size: 4096 2024-09-17T12:28:00.292 INFO:journalctl@ceph.mon.b.smithi196.stdout: partition_filters: 0 2024-09-17T12:28:00.292 INFO:journalctl@ceph.mon.b.smithi196.stdout: use_delta_encoding: 1 2024-09-17T12:28:00.292 INFO:journalctl@ceph.mon.b.smithi196.stdout: filter_policy: bloomfilter 2024-09-17T12:28:00.293 INFO:journalctl@ceph.mon.b.smithi196.stdout: whole_key_filtering: 1 2024-09-17T12:28:00.293 INFO:journalctl@ceph.mon.b.smithi196.stdout: verify_compression: 0 2024-09-17T12:28:00.293 INFO:journalctl@ceph.mon.b.smithi196.stdout: read_amp_bytes_per_bit: 0 2024-09-17T12:28:00.293 INFO:journalctl@ceph.mon.b.smithi196.stdout: format_version: 5 2024-09-17T12:28:00.293 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_index_compression: 1 2024-09-17T12:28:00.293 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_align: 0 2024-09-17T12:28:00.293 INFO:journalctl@ceph.mon.b.smithi196.stdout: max_auto_readahead_size: 262144 2024-09-17T12:28:00.293 INFO:journalctl@ceph.mon.b.smithi196.stdout: prepopulate_block_cache: 0 2024-09-17T12:28:00.293 INFO:journalctl@ceph.mon.b.smithi196.stdout: initial_auto_readahead_size: 8192 2024-09-17T12:28:00.293 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T12:28:00.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T12:28:00.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T12:28:00.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compression: NoCompression 2024-09-17T12:28:00.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T12:28:00.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T12:28:00.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T12:28:00.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.num_levels: 7 2024-09-17T12:28:00.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T12:28:00.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T12:28:00.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T12:28:00.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T12:28:00.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T12:28:00.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T12:28:00.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T12:28:00.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T12:28:00.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T12:28:00.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T12:28:00.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T12:28:00.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T12:28:00.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T12:28:00.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T12:28:00.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T12:28:00.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T12:28:00.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T12:28:00.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T12:28:00.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T12:28:00.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T12:28:00.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T12:28:00.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T12:28:00.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T12:28:00.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T12:28:00.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T12:28:00.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T12:28:00.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T12:28:00.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T12:28:00.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T12:28:00.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T12:28:00.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T12:28:00.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T12:28:00.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T12:28:00.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T12:28:00.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T12:28:00.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T12:28:00.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T12:28:00.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T12:28:00.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T12:28:00.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T12:28:00.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T12:28:00.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T12:28:00.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T12:28:00.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T12:28:00.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T12:28:00.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T12:28:00.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T12:28:00.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T12:28:00.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T12:28:00.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T12:28:00.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T12:28:00.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T12:28:00.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T12:28:00.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T12:28:00.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.inplace_update_support: 0 2024-09-17T12:28:00.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T12:28:00.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T12:28:00.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T12:28:00.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T12:28:00.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.bloom_locality: 0 2024-09-17T12:28:00.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.max_successive_merges: 0 2024-09-17T12:28:00.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T12:28:00.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T12:28:00.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T12:28:00.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T12:28:00.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.ttl: 2592000 2024-09-17T12:28:00.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T12:28:00.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T12:28:00.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T12:28:00.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.enable_blob_files: false 2024-09-17T12:28:00.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.min_blob_size: 0 2024-09-17T12:28:00.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T12:28:00.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T12:28:00.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T12:28:00.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T12:28:00.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T12:28:00.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T12:28:00.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T12:28:00.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T12:28:00.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T12:28:00.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 13039, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-09-17T12:28:00.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-17T12:28:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 566be203-b006-41f1-ba60-2c82ca3e7321 2024-09-17T12:28:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726576079831729, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-17T12:28:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-17T12:28:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726576079847983, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 4288735, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 13045, "largest_seqno": 14491, "table_properties": {"data_size": 4281641, "index_size": 4449, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1605, "raw_key_size": 15011, "raw_average_key_size": 23, "raw_value_size": 4267806, "raw_average_value_size": 6742, "num_data_blocks": 202, "num_entries": 633, "num_filter_entries": 633, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726576079, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "566be203-b006-41f1-ba60-2c82ca3e7321", "db_session_id": "PXREHE832XXUXVWGIZEV", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-17T12:28:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726576079848113, "job": 1, "event": "recovery_finished"} 2024-09-17T12:28:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-17T12:28:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T12:28:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T12:28:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558420802e00 2024-09-17T12:28:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: DB pointer 0x558420904000 2024-09-17T12:28:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T12:28:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T12:28:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** DB Stats ** 2024-09-17T12:28:00.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T12:28:00.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T12:28:00.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T12:28:00.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T12:28:00.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T12:28:00.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T12:28:00.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T12:28:00.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-17T12:28:00.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-17T12:28:00.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T12:28:00.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T12:28:00.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: L0 1/0 4.09 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 544.9 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-17T12:28:00.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: L6 1/0 8.38 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-17T12:28:00.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: Sum 2/0 12.47 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 544.9 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-17T12:28:00.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 544.9 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-17T12:28:00.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-17T12:28:00.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-17T12:28:00.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T12:28:00.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T12:28:00.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 544.9 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-17T12:28:00.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-17T12:28:00.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T12:28:00.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-17T12:28:00.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T12:28:00.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-09-17T12:28:00.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T12:28:00.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T12:28:00.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T12:28:00.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T12:28:00.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative compaction: 0.00 GB write, 198.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T12:28:00.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval compaction: 0.00 GB write, 198.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T12:28:00.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T12:28:00.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache BinnedLRUCache@0x5584207d3610#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-09-17T12:28:00.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-09-17T12:28:00.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-17T12:28:00.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T12:28:00.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: starting mon.b rank 1 at public addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] at bind addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:28:00.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: mon.b@-1(???) e3 preinit fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:28:00.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: mon.b@-1(???).mds e1 new map 2024-09-17T12:28:00.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: mon.b@-1(???).mds e1 print_map 2024-09-17T12:28:00.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: e1 2024-09-17T12:28:00.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-17T12:28:00.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T12:28:00.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T12:28:00.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: legacy client fscid: -1 2024-09-17T12:28:00.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-17T12:28:00.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: No filesystems configured 2024-09-17T12:28:00.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: mon.b@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-17T12:28:00.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: mon.b@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T12:28:00.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: mon.b@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T12:28:00.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: mon.b@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T12:28:00.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 ceph-mon[80339]: mon.b@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-09-17T12:28:00.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 bash[80320]: dc48dfc9e40cb61472b185c889f77662327130265661fdd9ef1d75eaf04ceb5a 2024-09-17T12:28:00.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:27:59 smithi196 systemd[1]: Started Ceph mon.b for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:28:00.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:00 smithi016 ceph-mon[87889]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:00.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:00 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T12:28:00.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:00 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T12:28:00.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:00 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T12:28:00.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:00 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-17T12:28:00.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:00 smithi016 ceph-mon[89784]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:00.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:00 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T12:28:00.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:00 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T12:28:00.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:00 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T12:28:00.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:00 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-17T12:28:01.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:00 smithi196 ceph-mon[80339]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:00 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T12:28:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:00 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T12:28:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:00 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T12:28:01.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:00 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-17T12:28:01.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:28:01.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:01.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T12:28:01.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: mgrmap e41: y(active, since 27s), standbys: x 2024-09-17T12:28:01.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: overall HEALTH_OK 2024-09-17T12:28:01.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T12:28:01.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: mon.b calling monitor election 2024-09-17T12:28:01.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: mon.a calling monitor election 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: mon.c calling monitor election 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: monmap epoch 4 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: last_changed 2024-09-17T12:28:00.548067+0000 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: created 2024-09-17T12:15:22.758286+0000 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: min_mon_release 19 (squid) 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: election_strategy: 1 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: 2: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-09-17T12:28:01.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: fsmap 2024-09-17T12:28:01.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T12:28:01.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: mgrmap e41: y(active, since 27s), standbys: x 2024-09-17T12:28:01.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: overall HEALTH_OK 2024-09-17T12:28:01.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='' 2024-09-17T12:28:01.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:01.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:01 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:28:01.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:28:01.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:01.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T12:28:01.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: mgrmap e41: y(active, since 27s), standbys: x 2024-09-17T12:28:01.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: overall HEALTH_OK 2024-09-17T12:28:01.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T12:28:01.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: mon.b calling monitor election 2024-09-17T12:28:01.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: mon.a calling monitor election 2024-09-17T12:28:01.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: mon.c calling monitor election 2024-09-17T12:28:01.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: monmap epoch 4 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: last_changed 2024-09-17T12:28:00.548067+0000 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: created 2024-09-17T12:15:22.758286+0000 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: min_mon_release 19 (squid) 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: election_strategy: 1 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: 2: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: fsmap 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T12:28:01.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: mgrmap e41: y(active, since 27s), standbys: x 2024-09-17T12:28:01.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: overall HEALTH_OK 2024-09-17T12:28:01.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='' 2024-09-17T12:28:01.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:01.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:28:01.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-17T12:28:01.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:01.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T12:28:01.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: mgrmap e41: y(active, since 27s), standbys: x 2024-09-17T12:28:01.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: overall HEALTH_OK 2024-09-17T12:28:01.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T12:28:01.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: mon.b calling monitor election 2024-09-17T12:28:01.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: mon.a calling monitor election 2024-09-17T12:28:01.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: mon.c calling monitor election 2024-09-17T12:28:01.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T12:28:01.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T12:28:01.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T12:28:01.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: monmap epoch 4 2024-09-17T12:28:01.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:28:01.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: last_changed 2024-09-17T12:28:00.548067+0000 2024-09-17T12:28:01.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: created 2024-09-17T12:15:22.758286+0000 2024-09-17T12:28:01.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: min_mon_release 19 (squid) 2024-09-17T12:28:01.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: election_strategy: 1 2024-09-17T12:28:01.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-17T12:28:01.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-17T12:28:01.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: 2: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-09-17T12:28:01.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: fsmap 2024-09-17T12:28:01.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T12:28:01.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: mgrmap e41: y(active, since 27s), standbys: x 2024-09-17T12:28:01.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: overall HEALTH_OK 2024-09-17T12:28:01.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='' 2024-09-17T12:28:01.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:01.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:01 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:28:02.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:02 smithi196 ceph-mon[80339]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:02.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:02 smithi016 ceph-mon[87889]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:02.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:02 smithi016 ceph-mon[89784]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:04 smithi016 ceph-mon[87889]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:04 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:28:04.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:04 smithi016 ceph-mon[89784]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:04.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:04 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:28:04.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:04 smithi196 ceph-mon[80339]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:04.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:04 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:28:05.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:28:04 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:28:04.859+0000 7f9d37c73640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-17T12:28:05.875 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:05 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:05.875 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:05 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:05.875 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:05 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:05.875 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:05 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:05.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:05 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:05.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:05 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:05.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:05 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:05.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:05 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:05.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:05 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:05.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:05 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:05.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:05 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:05.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:05 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:06.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:28:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:28:06] "GET /metrics HTTP/1.1" 200 38429 "" "Prometheus/2.43.0" 2024-09-17T12:28:06.501 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:28:07.085 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:28:07.085 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (6m) 15s ago 7m 25.3M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:28:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (6m) 2s ago 6m 89.2M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:28:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (61s) 15s ago 5m 55.9M - 3.8 980a0bfcd684 843db1407344 2024-09-17T12:28:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (53s) 2s ago 10m 453M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:28:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (85s) 15s ago 12m 503M - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:28:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (40s) 15s ago 12m 38.9M 2048M 19.3.0-4937-g74984932 6c1fcab85602 4a83b787c605 2024-09-17T12:28:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (7s) 2s ago 11m 24.7M 2048M 19.3.0-4937-g74984932 6c1fcab85602 dc48dfc9e40c 2024-09-17T12:28:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (20s) 15s ago 11m 22.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 aa5074acd22a 2024-09-17T12:28:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (6m) 15s ago 6m 15.8M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:28:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (6m) 2s ago 6m 15.4M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:28:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (10m) 15s ago 10m 61.5M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 e30aaffd3b5f 2024-09-17T12:28:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (10m) 15s ago 10m 54.9M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 3ad94c30c9c9 2024-09-17T12:28:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (9m) 15s ago 9m 56.2M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 1fa310695876 2024-09-17T12:28:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (9m) 15s ago 9m 60.4M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 4a4f8eeaff13 2024-09-17T12:28:07.087 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (9m) 2s ago 9m 55.7M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 afa20056b398 2024-09-17T12:28:07.087 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (8m) 2s ago 8m 51.2M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 f0dcabb8902f 2024-09-17T12:28:07.087 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (8m) 2s ago 8m 60.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:28:07.087 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (7m) 2s ago 7m 61.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:28:07.087 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (58s) 2s ago 7m 42.7M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:28:07.087 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (5m) 15s ago 5m 79.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:28:07.087 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (5m) 2s ago 5m 80.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:28:07.087 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (5m) 15s ago 5m 81.7M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:28:07.087 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (5m) 2s ago 5m 79.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:28:07.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:07 smithi016 ceph-mon[87889]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:07.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:07 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:07.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:07 smithi016 ceph-mon[87889]: from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:07.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:07 smithi016 ceph-mon[87889]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:07.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:07 smithi016 ceph-mon[87889]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:07.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:07 smithi016 ceph-mon[89784]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:07.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:07 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:07.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:07 smithi016 ceph-mon[89784]: from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:07.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:07 smithi016 ceph-mon[89784]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:07.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:07 smithi016 ceph-mon[89784]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:07.443 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:28:07.443 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:28:07.443 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:28:07.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:28:07.443 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:28:07.443 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:28:07.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:28:07.444 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:28:07.444 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T12:28:07.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:28:07.444 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:28:07.444 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:28:07.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:28:07.444 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:28:07.444 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 12, 2024-09-17T12:28:07.444 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 5 2024-09-17T12:28:07.444 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:28:07.444 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:28:07.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:07 smithi196 ceph-mon[80339]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:07.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:07 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:07.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:07 smithi196 ceph-mon[80339]: from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:07.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:07 smithi196 ceph-mon[80339]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:07.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:07 smithi196 ceph-mon[80339]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:07.744 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:28:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:28:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:28:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:28:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:28:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-09-17T12:28:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:28:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:28:07.745 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "5/23 daemons upgraded", 2024-09-17T12:28:07.745 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading mon daemons", 2024-09-17T12:28:07.745 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:28:07.745 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:28:08.096 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-09-17T12:28:08.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:08 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/3773584279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:08.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:08 smithi016 ceph-mon[89784]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:08.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:08 smithi016 ceph-mon[89784]: from='client.44126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:08.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:08 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/1932410611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:28:08.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:08 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/3773584279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:08.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:08 smithi016 ceph-mon[87889]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:08.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:08 smithi016 ceph-mon[87889]: from='client.44126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:08.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:08 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/1932410611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:28:08.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:08 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/3773584279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:08.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:08 smithi196 ceph-mon[80339]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:08.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:08 smithi196 ceph-mon[80339]: from='client.44126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:08.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:08 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/1932410611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:28:09.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:09.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:09.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:09.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:28:09.649 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:09.649 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[87889]: Reconfiguring mon.a (monmap changed)... 2024-09-17T12:28:09.649 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T12:28:09.649 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T12:28:09.649 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:09.649 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[87889]: Reconfiguring daemon mon.a on smithi016 2024-09-17T12:28:09.649 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:09.649 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:09.649 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:09.649 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:28:09.649 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:09.650 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[89784]: Reconfiguring mon.a (monmap changed)... 2024-09-17T12:28:09.650 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T12:28:09.650 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T12:28:09.650 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:09.650 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:09 smithi016 ceph-mon[89784]: Reconfiguring daemon mon.a on smithi016 2024-09-17T12:28:09.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:09 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:09 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:09 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:09 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:28:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:09 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:09 smithi196 ceph-mon[80339]: Reconfiguring mon.a (monmap changed)... 2024-09-17T12:28:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:09 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T12:28:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:09 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T12:28:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:09 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:09 smithi196 ceph-mon[80339]: Reconfiguring daemon mon.a on smithi016 2024-09-17T12:28:11.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:10 smithi196 ceph-mon[80339]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:11.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:10 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:11.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:10 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:11.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:10 smithi196 ceph-mon[80339]: Reconfiguring mgr.y (monmap changed)... 2024-09-17T12:28:11.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:10 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T12:28:11.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:10 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T12:28:11.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:10 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-17T12:28:11.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:10 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:11.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:10 smithi196 ceph-mon[80339]: Reconfiguring daemon mgr.y on smithi016 2024-09-17T12:28:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[87889]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[87889]: Reconfiguring mgr.y (monmap changed)... 2024-09-17T12:28:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T12:28:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T12:28:11.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-17T12:28:11.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:11.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[87889]: Reconfiguring daemon mgr.y on smithi016 2024-09-17T12:28:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[89784]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[89784]: Reconfiguring mgr.y (monmap changed)... 2024-09-17T12:28:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T12:28:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T12:28:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-17T12:28:11.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:11.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:10 smithi016 ceph-mon[89784]: Reconfiguring daemon mgr.y on smithi016 2024-09-17T12:28:12.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:12.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:12.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[87889]: Reconfiguring mon.c (monmap changed)... 2024-09-17T12:28:12.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T12:28:12.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T12:28:12.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:12.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[87889]: Reconfiguring daemon mon.c on smithi016 2024-09-17T12:28:12.120 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:12.120 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:12.120 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[89784]: Reconfiguring mon.c (monmap changed)... 2024-09-17T12:28:12.120 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T12:28:12.121 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T12:28:12.121 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:12.121 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:11 smithi016 ceph-mon[89784]: Reconfiguring daemon mon.c on smithi016 2024-09-17T12:28:12.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:11 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:11 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:11 smithi196 ceph-mon[80339]: Reconfiguring mon.c (monmap changed)... 2024-09-17T12:28:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:11 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T12:28:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:11 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T12:28:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:11 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:11 smithi196 ceph-mon[80339]: Reconfiguring daemon mon.c on smithi016 2024-09-17T12:28:13.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[87889]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:13.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:13.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:13.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[87889]: Reconfiguring osd.0 (monmap changed)... 2024-09-17T12:28:13.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-17T12:28:13.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:13.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[87889]: Reconfiguring daemon osd.0 on smithi016 2024-09-17T12:28:13.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[89784]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:13.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:13.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:13.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[89784]: Reconfiguring osd.0 (monmap changed)... 2024-09-17T12:28:13.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-17T12:28:13.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:13.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:13 smithi016 ceph-mon[89784]: Reconfiguring daemon osd.0 on smithi016 2024-09-17T12:28:13.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:13 smithi196 ceph-mon[80339]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:13.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:13 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:13.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:13 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:13.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:13 smithi196 ceph-mon[80339]: Reconfiguring osd.0 (monmap changed)... 2024-09-17T12:28:13.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:13 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-17T12:28:13.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:13 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:13.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:13 smithi196 ceph-mon[80339]: Reconfiguring daemon osd.0 on smithi016 2024-09-17T12:28:14.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:14 smithi196 ceph-mon[80339]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:14.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:14 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:14.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:14 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:14.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:14 smithi196 ceph-mon[80339]: Reconfiguring osd.1 (monmap changed)... 2024-09-17T12:28:14.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:14 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-17T12:28:14.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:14 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:14.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:14 smithi196 ceph-mon[80339]: Reconfiguring daemon osd.1 on smithi016 2024-09-17T12:28:14.824 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[87889]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:14.824 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:14.824 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:14.824 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[87889]: Reconfiguring osd.1 (monmap changed)... 2024-09-17T12:28:14.824 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-17T12:28:14.824 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:14.824 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[87889]: Reconfiguring daemon osd.1 on smithi016 2024-09-17T12:28:14.825 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[89784]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:14.825 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:14.825 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:14.825 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[89784]: Reconfiguring osd.1 (monmap changed)... 2024-09-17T12:28:14.825 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-17T12:28:14.825 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:14.825 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:14 smithi016 ceph-mon[89784]: Reconfiguring daemon osd.1 on smithi016 2024-09-17T12:28:16.189 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:16.189 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:16.189 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:15 smithi016 ceph-mon[87889]: Reconfiguring osd.2 (monmap changed)... 2024-09-17T12:28:16.189 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-17T12:28:16.189 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:16.189 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:15 smithi016 ceph-mon[87889]: Reconfiguring daemon osd.2 on smithi016 2024-09-17T12:28:16.190 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:16.190 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:16.190 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:15 smithi016 ceph-mon[89784]: Reconfiguring osd.2 (monmap changed)... 2024-09-17T12:28:16.190 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-17T12:28:16.190 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:16.190 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:15 smithi016 ceph-mon[89784]: Reconfiguring daemon osd.2 on smithi016 2024-09-17T12:28:16.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:16.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:16.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:15 smithi196 ceph-mon[80339]: Reconfiguring osd.2 (monmap changed)... 2024-09-17T12:28:16.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-17T12:28:16.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:16.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:15 smithi196 ceph-mon[80339]: Reconfiguring daemon osd.2 on smithi016 2024-09-17T12:28:16.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:28:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:28:16] "GET /metrics HTTP/1.1" 200 38429 "" "Prometheus/2.43.0" 2024-09-17T12:28:17.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:16 smithi016 ceph-mon[87889]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:17.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:16 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:17.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:16 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:17.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:16 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:17.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:16 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-17T12:28:17.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:16 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:17.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:16 smithi016 ceph-mon[89784]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:17.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:16 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:17.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:16 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:17.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:16 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:17.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:16 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-17T12:28:17.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:16 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:16 smithi196 ceph-mon[80339]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:16 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:16 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:16 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:16 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-17T12:28:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:16 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:18.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[89784]: Reconfiguring osd.3 (monmap changed)... 2024-09-17T12:28:18.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[89784]: Reconfiguring daemon osd.3 on smithi016 2024-09-17T12:28:18.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:18.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:18.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:18.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:18.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:18.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[87889]: Reconfiguring osd.3 (monmap changed)... 2024-09-17T12:28:18.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[87889]: Reconfiguring daemon osd.3 on smithi016 2024-09-17T12:28:18.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:18.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:18.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:18.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:18.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:17 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:18.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:17 smithi196 ceph-mon[80339]: Reconfiguring osd.3 (monmap changed)... 2024-09-17T12:28:18.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:17 smithi196 ceph-mon[80339]: Reconfiguring daemon osd.3 on smithi016 2024-09-17T12:28:18.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:17 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:18.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:17 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:18.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:17 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:18.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:17 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:18.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:17 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:19.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[87889]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:19.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[87889]: Reconfiguring rgw.foo.smithi016.tquwud (monmap changed)... 2024-09-17T12:28:19.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[87889]: Reconfiguring daemon rgw.foo.smithi016.tquwud on smithi016 2024-09-17T12:28:19.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:19.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:19.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:19.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:19.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:19.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:28:19.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[89784]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:19.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[89784]: Reconfiguring rgw.foo.smithi016.tquwud (monmap changed)... 2024-09-17T12:28:19.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[89784]: Reconfiguring daemon rgw.foo.smithi016.tquwud on smithi016 2024-09-17T12:28:19.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:19.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:19.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:19.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:19.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:19.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:18 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:28:19.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:18 smithi196 ceph-mon[80339]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:18 smithi196 ceph-mon[80339]: Reconfiguring rgw.foo.smithi016.tquwud (monmap changed)... 2024-09-17T12:28:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:18 smithi196 ceph-mon[80339]: Reconfiguring daemon rgw.foo.smithi016.tquwud on smithi016 2024-09-17T12:28:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:18 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:18 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:18 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:18 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:18 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:18 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:28:20.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:19 smithi196 ceph-mon[80339]: Reconfiguring rgw.smpl.smithi016.qdkyfn (monmap changed)... 2024-09-17T12:28:20.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:19 smithi196 ceph-mon[80339]: Reconfiguring daemon rgw.smpl.smithi016.qdkyfn on smithi016 2024-09-17T12:28:20.290 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:19 smithi016 ceph-mon[87889]: Reconfiguring rgw.smpl.smithi016.qdkyfn (monmap changed)... 2024-09-17T12:28:20.290 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:19 smithi016 ceph-mon[87889]: Reconfiguring daemon rgw.smpl.smithi016.qdkyfn on smithi016 2024-09-17T12:28:20.290 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:19 smithi016 ceph-mon[89784]: Reconfiguring rgw.smpl.smithi016.qdkyfn (monmap changed)... 2024-09-17T12:28:20.291 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:19 smithi016 ceph-mon[89784]: Reconfiguring daemon rgw.smpl.smithi016.qdkyfn on smithi016 2024-09-17T12:28:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:20 smithi196 ceph-mon[80339]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:20 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:20 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:20 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T12:28:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:20 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T12:28:20.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:20 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:21.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:20 smithi016 ceph-mon[87889]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:21.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:20 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:21.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:20 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:21.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:20 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T12:28:21.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:20 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T12:28:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:20 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:21.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:20 smithi016 ceph-mon[89784]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:21.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:20 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:21.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:20 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:21.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:20 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T12:28:21.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:20 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T12:28:21.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:20 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:22.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:21 smithi196 ceph-mon[80339]: Reconfiguring mon.b (monmap changed)... 2024-09-17T12:28:22.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:21 smithi196 ceph-mon[80339]: Reconfiguring daemon mon.b on smithi196 2024-09-17T12:28:22.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:21 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:22.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:21 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:22.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:21 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T12:28:22.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:21 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T12:28:22.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:21 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-17T12:28:22.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:21 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:22.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[89784]: Reconfiguring mon.b (monmap changed)... 2024-09-17T12:28:22.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[89784]: Reconfiguring daemon mon.b on smithi196 2024-09-17T12:28:22.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:22.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:22.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T12:28:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T12:28:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-17T12:28:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:22.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[87889]: Reconfiguring mon.b (monmap changed)... 2024-09-17T12:28:22.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[87889]: Reconfiguring daemon mon.b on smithi196 2024-09-17T12:28:22.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:22.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:22.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T12:28:22.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T12:28:22.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-17T12:28:22.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:21 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:23.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:22 smithi196 ceph-mon[80339]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:23.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:22 smithi196 ceph-mon[80339]: Reconfiguring mgr.x (monmap changed)... 2024-09-17T12:28:23.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:22 smithi196 ceph-mon[80339]: Reconfiguring daemon mgr.x on smithi196 2024-09-17T12:28:23.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:22 smithi016 ceph-mon[87889]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:23.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:22 smithi016 ceph-mon[87889]: Reconfiguring mgr.x (monmap changed)... 2024-09-17T12:28:23.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:22 smithi016 ceph-mon[87889]: Reconfiguring daemon mgr.x on smithi196 2024-09-17T12:28:23.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:22 smithi016 ceph-mon[89784]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:23.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:22 smithi016 ceph-mon[89784]: Reconfiguring mgr.x (monmap changed)... 2024-09-17T12:28:23.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:22 smithi016 ceph-mon[89784]: Reconfiguring daemon mgr.x on smithi196 2024-09-17T12:28:24.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:24 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:24.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:24 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:24.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:24 smithi016 ceph-mon[87889]: Reconfiguring osd.4 (monmap changed)... 2024-09-17T12:28:24.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:24 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-17T12:28:24.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:24 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:24.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:24 smithi016 ceph-mon[87889]: Reconfiguring daemon osd.4 on smithi196 2024-09-17T12:28:24.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:24 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:24.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:24 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:24.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:24 smithi016 ceph-mon[89784]: Reconfiguring osd.4 (monmap changed)... 2024-09-17T12:28:24.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:24 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-17T12:28:24.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:24 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:24.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:24 smithi016 ceph-mon[89784]: Reconfiguring daemon osd.4 on smithi196 2024-09-17T12:28:24.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:24 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:24.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:24 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:24.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:24 smithi196 ceph-mon[80339]: Reconfiguring osd.4 (monmap changed)... 2024-09-17T12:28:24.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:24 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-17T12:28:24.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:24 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:24.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:24 smithi196 ceph-mon[80339]: Reconfiguring daemon osd.4 on smithi196 2024-09-17T12:28:25.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:25 smithi016 ceph-mon[87889]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:25.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:25 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:25.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:25 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:25.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:25 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-17T12:28:25.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:25 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:25.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:25 smithi016 ceph-mon[89784]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:25.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:25 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:25.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:25 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:25.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:25 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-17T12:28:25.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:25 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:25.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:25 smithi196 ceph-mon[80339]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:25.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:25 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:25.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:25 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:25.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:25 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-17T12:28:25.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:25 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:26.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:28:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:28:26] "GET /metrics HTTP/1.1" 200 38500 "" "Prometheus/2.43.0" 2024-09-17T12:28:26.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:26 smithi016 ceph-mon[87889]: Reconfiguring osd.5 (monmap changed)... 2024-09-17T12:28:26.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:26 smithi016 ceph-mon[87889]: Reconfiguring daemon osd.5 on smithi196 2024-09-17T12:28:26.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:26 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:26.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:26 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:26.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:26 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-17T12:28:26.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:26 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:26.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:26 smithi016 ceph-mon[89784]: Reconfiguring osd.5 (monmap changed)... 2024-09-17T12:28:26.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:26 smithi016 ceph-mon[89784]: Reconfiguring daemon osd.5 on smithi196 2024-09-17T12:28:26.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:26 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:26.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:26 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:26.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:26 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-17T12:28:26.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:26 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:26.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:26 smithi196 ceph-mon[80339]: Reconfiguring osd.5 (monmap changed)... 2024-09-17T12:28:26.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:26 smithi196 ceph-mon[80339]: Reconfiguring daemon osd.5 on smithi196 2024-09-17T12:28:26.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:26 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:26.500 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:26 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:26.500 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:26 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-17T12:28:26.500 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:26 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:27.253 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:27 smithi196 ceph-mon[80339]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:27.253 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:27 smithi196 ceph-mon[80339]: Reconfiguring osd.6 (monmap changed)... 2024-09-17T12:28:27.253 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:27 smithi196 ceph-mon[80339]: Reconfiguring daemon osd.6 on smithi196 2024-09-17T12:28:27.253 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:27 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:27.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:27 smithi016 ceph-mon[87889]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:27.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:27 smithi016 ceph-mon[87889]: Reconfiguring osd.6 (monmap changed)... 2024-09-17T12:28:27.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:27 smithi016 ceph-mon[87889]: Reconfiguring daemon osd.6 on smithi196 2024-09-17T12:28:27.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:27 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:27.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:27 smithi016 ceph-mon[89784]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:27.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:27 smithi016 ceph-mon[89784]: Reconfiguring osd.6 (monmap changed)... 2024-09-17T12:28:27.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:27 smithi016 ceph-mon[89784]: Reconfiguring daemon osd.6 on smithi196 2024-09-17T12:28:27.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:27 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:28.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:28.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[87889]: Reconfiguring osd.7 (monmap changed)... 2024-09-17T12:28:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-17T12:28:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[87889]: Reconfiguring daemon osd.7 on smithi196 2024-09-17T12:28:28.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[87889]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:28.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:28.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:28.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[89784]: Reconfiguring osd.7 (monmap changed)... 2024-09-17T12:28:28.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-17T12:28:28.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:28.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[89784]: Reconfiguring daemon osd.7 on smithi196 2024-09-17T12:28:28.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:28 smithi016 ceph-mon[89784]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:28.710 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:28 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:28.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:28 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:28.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:28 smithi196 ceph-mon[80339]: Reconfiguring osd.7 (monmap changed)... 2024-09-17T12:28:28.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:28 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-17T12:28:28.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:28 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:28.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:28 smithi196 ceph-mon[80339]: Reconfiguring daemon osd.7 on smithi196 2024-09-17T12:28:28.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:28 smithi196 ceph-mon[80339]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:30.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:29 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:30.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:29 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:30.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:29 smithi196 ceph-mon[80339]: Reconfiguring rgw.foo.smithi196.wvbocw (monmap changed)... 2024-09-17T12:28:30.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:29 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:30.178 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:29 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:30.178 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:29 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:30.178 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:29 smithi196 ceph-mon[80339]: Reconfiguring daemon rgw.foo.smithi196.wvbocw on smithi196 2024-09-17T12:28:30.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:30.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:30.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[87889]: Reconfiguring rgw.foo.smithi196.wvbocw (monmap changed)... 2024-09-17T12:28:30.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:30.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:30.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:30.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[87889]: Reconfiguring daemon rgw.foo.smithi196.wvbocw on smithi196 2024-09-17T12:28:30.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:30.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:30.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[89784]: Reconfiguring rgw.foo.smithi196.wvbocw (monmap changed)... 2024-09-17T12:28:30.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:30.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:30.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:30.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:29 smithi016 ceph-mon[89784]: Reconfiguring daemon rgw.foo.smithi196.wvbocw on smithi196 2024-09-17T12:28:31.515 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:31 smithi196 ceph-mon[80339]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:31.515 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:31 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:31.515 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:31 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:31.516 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:31 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:31.516 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:31 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:31.516 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:31 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:31 smithi016 ceph-mon[87889]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:31 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:31 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:31 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:31 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:31 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:31.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:31 smithi016 ceph-mon[89784]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:31.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:31 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:31.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:31 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:31.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:31 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:31.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:31 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:28:31.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:31 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:32.258 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: Reconfiguring rgw.smpl.smithi196.ghtvuz (monmap changed)... 2024-09-17T12:28:32.258 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: Reconfiguring daemon rgw.smpl.smithi196.ghtvuz on smithi196 2024-09-17T12:28:32.259 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:32.259 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:32.259 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:28:32.259 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:32.259 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:32.259 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:32.259 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-17T12:28:32.259 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-17T12:28:32.259 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-17T12:28:32.259 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-17T12:28:32.260 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-17T12:28:32.260 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-17T12:28:32.260 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-17T12:28:32.260 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-17T12:28:32.260 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-17T12:28:32.260 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:32.260 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:32.260 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-17T12:28:32.260 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: Reconfiguring rgw.smpl.smithi196.ghtvuz (monmap changed)... 2024-09-17T12:28:32.261 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: Reconfiguring daemon rgw.smpl.smithi196.ghtvuz on smithi196 2024-09-17T12:28:32.261 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:32.261 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:32.261 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:28:32.261 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:32.261 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:32.261 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:32.261 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-17T12:28:32.261 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-17T12:28:32.261 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-17T12:28:32.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-17T12:28:32.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-17T12:28:32.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-17T12:28:32.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-17T12:28:32.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-17T12:28:32.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-17T12:28:32.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:32.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:32.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:32 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-17T12:28:32.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: Reconfiguring rgw.smpl.smithi196.ghtvuz (monmap changed)... 2024-09-17T12:28:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: Reconfiguring daemon rgw.smpl.smithi196.ghtvuz on smithi196 2024-09-17T12:28:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:28:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-17T12:28:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-17T12:28:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-17T12:28:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-17T12:28:32.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-17T12:28:32.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-17T12:28:32.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-17T12:28:32.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-17T12:28:32.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-17T12:28:32.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:32.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:32.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:32 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-17T12:28:33.490 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[87889]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:33.491 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[87889]: Upgrade: Setting container_image for all mon 2024-09-17T12:28:33.491 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[87889]: Upgrade: Setting container_image for all crash 2024-09-17T12:28:33.491 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T12:28:33.491 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[87889]: Upgrade: osd.0 is safe to restart 2024-09-17T12:28:33.491 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:33.491 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-17T12:28:33.491 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:33.491 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[89784]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:33.491 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[89784]: Upgrade: Setting container_image for all mon 2024-09-17T12:28:33.491 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[89784]: Upgrade: Setting container_image for all crash 2024-09-17T12:28:33.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T12:28:33.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[89784]: Upgrade: osd.0 is safe to restart 2024-09-17T12:28:33.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:33.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-17T12:28:33.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:33 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:33.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:33 smithi196 ceph-mon[80339]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:33.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:33 smithi196 ceph-mon[80339]: Upgrade: Setting container_image for all mon 2024-09-17T12:28:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:33 smithi196 ceph-mon[80339]: Upgrade: Setting container_image for all crash 2024-09-17T12:28:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:33 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T12:28:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:33 smithi196 ceph-mon[80339]: Upgrade: osd.0 is safe to restart 2024-09-17T12:28:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:33 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:33 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-17T12:28:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:33 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:34.341 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:34 smithi016 ceph-mon[87889]: Upgrade: Updating osd.0 2024-09-17T12:28:34.341 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:34 smithi016 ceph-mon[87889]: Deploying daemon osd.0 on smithi016 2024-09-17T12:28:34.341 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:28:34.342 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:34 smithi016 ceph-mon[89784]: Upgrade: Updating osd.0 2024-09-17T12:28:34.342 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:34 smithi016 ceph-mon[89784]: Deploying daemon osd.0 on smithi016 2024-09-17T12:28:34.342 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:34 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:28:34.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:34 smithi196 ceph-mon[80339]: Upgrade: Updating osd.0 2024-09-17T12:28:34.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:34 smithi196 ceph-mon[80339]: Deploying daemon osd.0 on smithi016 2024-09-17T12:28:34.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:34 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:28:35.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:35 smithi196 ceph-mon[80339]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:35.603 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:35 smithi016 ceph-mon[89784]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:35.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:35 smithi016 ceph-mon[87889]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:28:35.872 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:35 smithi016 systemd[1]: Stopping Ceph osd.0 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:28:36.191 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0[38071]: 2024-09-17T12:28:35.899+0000 7f77c5d2c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:28:36.191 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0[38071]: 2024-09-17T12:28:35.899+0000 7f77c5d2c640 -1 osd.0 104 *** Got signal Terminated *** 2024-09-17T12:28:36.191 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:35 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0[38071]: 2024-09-17T12:28:35.899+0000 7f77c5d2c640 -1 osd.0 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:28:36.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:36 smithi196 ceph-mon[80339]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:36.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:36 smithi196 ceph-mon[80339]: osd.0 marked itself down and dead 2024-09-17T12:28:36.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:36 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:36.594 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:36 smithi016 ceph-mon[87889]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:36.594 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:36 smithi016 ceph-mon[87889]: osd.0 marked itself down and dead 2024-09-17T12:28:36.594 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:36 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:36.594 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:28:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:28:36] "GET /metrics HTTP/1.1" 200 38502 "" "Prometheus/2.43.0" 2024-09-17T12:28:36.595 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:36 smithi016 ceph-mon[89784]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:36.595 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:36 smithi016 ceph-mon[89784]: osd.0 marked itself down and dead 2024-09-17T12:28:36.595 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:36 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:36.851 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:36 smithi016 podman[94349]: 2024-09-17 12:28:36.592912859 +0000 UTC m=+0.798088490 container died e30aaffd3b5f209ca0d1fa1b4776db49a25f740e1079da79c97f57df7eeaebc2 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-17T12:28:37.191 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:36 smithi016 podman[94349]: 2024-09-17 12:28:36.949824094 +0000 UTC m=+1.154999720 container cleanup e30aaffd3b5f209ca0d1fa1b4776db49a25f740e1079da79c97f57df7eeaebc2 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:28:37.191 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:36 smithi016 bash[94349]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0 2024-09-17T12:28:37.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:37 smithi196 ceph-mon[80339]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:28:37.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:37 smithi196 ceph-mon[80339]: osdmap e105: 8 total, 7 up, 8 in 2024-09-17T12:28:37.587 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:37 smithi016 ceph-mon[87889]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:28:37.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:37 smithi016 ceph-mon[87889]: osdmap e105: 8 total, 7 up, 8 in 2024-09-17T12:28:37.588 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:37 smithi016 ceph-mon[89784]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:28:37.588 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:37 smithi016 ceph-mon[89784]: osdmap e105: 8 total, 7 up, 8 in 2024-09-17T12:28:37.588 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:37 smithi016 podman[94375]: 2024-09-17 12:28:37.206705825 +0000 UTC m=+0.611126773 container remove e30aaffd3b5f209ca0d1fa1b4776db49a25f740e1079da79c97f57df7eeaebc2 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b) 2024-09-17T12:28:37.888 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:37 smithi016 podman[94465]: 2024-09-17 12:28:37.489061802 +0000 UTC m=+0.016614617 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:28:37.889 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:37 smithi016 podman[94465]: 2024-09-17 12:28:37.593759251 +0000 UTC m=+0.121312056 container create bbb2587f1a5fbab21433bfe687281eb2d0ee3f2c24538031bc6c7762e6adbe6a (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=main-7498493, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-17T12:28:37.889 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:37 smithi016 podman[94465]: 2024-09-17 12:28:37.812840297 +0000 UTC m=+0.340393113 container init bbb2587f1a5fbab21433bfe687281eb2d0ee3f2c24538031bc6c7762e6adbe6a (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=main-7498493) 2024-09-17T12:28:37.889 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:37 smithi016 podman[94465]: 2024-09-17 12:28:37.816176092 +0000 UTC m=+0.343728905 container start bbb2587f1a5fbab21433bfe687281eb2d0ee3f2c24538031bc6c7762e6adbe6a (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=main-7498493, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-17T12:28:37.889 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:37 smithi016 podman[94465]: 2024-09-17 12:28:37.887735006 +0000 UTC m=+0.415287814 container attach bbb2587f1a5fbab21433bfe687281eb2d0ee3f2c24538031bc6c7762e6adbe6a (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=main-7498493, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:28:38.191 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:38 smithi016 podman[94465]: 2024-09-17 12:28:38.014841586 +0000 UTC m=+0.542394394 container died bbb2587f1a5fbab21433bfe687281eb2d0ee3f2c24538031bc6c7762e6adbe6a (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=main-7498493, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-17T12:28:38.414 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:28:38.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:38 smithi196 ceph-mon[80339]: osdmap e106: 8 total, 7 up, 8 in 2024-09-17T12:28:38.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:38 smithi196 ceph-mon[80339]: pgmap v37: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T12:28:38.634 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:38 smithi016 ceph-mon[87889]: osdmap e106: 8 total, 7 up, 8 in 2024-09-17T12:28:38.635 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:38 smithi016 ceph-mon[87889]: pgmap v37: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T12:28:38.635 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:38 smithi016 ceph-mon[89784]: osdmap e106: 8 total, 7 up, 8 in 2024-09-17T12:28:38.635 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:38 smithi016 ceph-mon[89784]: pgmap v37: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T12:28:38.941 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:38 smithi016 podman[94489]: 2024-09-17 12:28:38.633355081 +0000 UTC m=+0.612453726 container remove bbb2587f1a5fbab21433bfe687281eb2d0ee3f2c24538031bc6c7762e6adbe6a (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=main-7498493, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-17T12:28:38.941 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:38 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.0.service: Deactivated successfully. 2024-09-17T12:28:38.941 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:38 smithi016 systemd[1]: Stopped Ceph osd.0 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:28:38.941 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:38 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.0.service: Consumed 7.269s CPU time. 2024-09-17T12:28:38.942 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:38 smithi016 systemd[1]: Starting Ceph osd.0 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:28:39.027 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:28:39.028 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (6m) 47s ago 7m 25.3M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:28:39.028 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (7m) 34s ago 7m 89.2M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:28:39.028 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (93s) 47s ago 5m 55.9M - 3.8 980a0bfcd684 843db1407344 2024-09-17T12:28:39.028 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (85s) 34s ago 11m 453M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:28:39.028 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (117s) 47s ago 13m 503M - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:28:39.028 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (72s) 47s ago 13m 38.9M 2048M 19.3.0-4937-g74984932 6c1fcab85602 4a83b787c605 2024-09-17T12:28:39.028 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (39s) 34s ago 11m 24.7M 2048M 19.3.0-4937-g74984932 6c1fcab85602 dc48dfc9e40c 2024-09-17T12:28:39.028 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (52s) 47s ago 11m 22.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 aa5074acd22a 2024-09-17T12:28:39.028 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (7m) 47s ago 7m 15.8M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:28:39.028 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (6m) 34s ago 6m 15.4M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:28:39.028 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (11m) 47s ago 11m 61.5M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 e30aaffd3b5f 2024-09-17T12:28:39.028 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (10m) 47s ago 10m 54.9M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 3ad94c30c9c9 2024-09-17T12:28:39.029 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (10m) 47s ago 10m 56.2M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 1fa310695876 2024-09-17T12:28:39.029 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (9m) 47s ago 9m 60.4M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 4a4f8eeaff13 2024-09-17T12:28:39.029 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (9m) 34s ago 9m 55.7M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 afa20056b398 2024-09-17T12:28:39.029 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (9m) 34s ago 9m 51.2M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 f0dcabb8902f 2024-09-17T12:28:39.029 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (8m) 34s ago 8m 60.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:28:39.029 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (8m) 34s ago 8m 61.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:28:39.029 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (90s) 34s ago 7m 42.7M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:28:39.029 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (6m) 47s ago 6m 79.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:28:39.029 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (6m) 34s ago 6m 80.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:28:39.029 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (6m) 47s ago 6m 81.7M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:28:39.029 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (6m) 34s ago 6m 79.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:28:39.338 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:39 smithi016 ceph-mon[87889]: from='client.34203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:39.338 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:39 smithi016 ceph-mon[87889]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:39.338 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:39 smithi016 ceph-mon[87889]: from='client.44150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:39.339 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:39 smithi016 podman[94656]: 2024-09-17 12:28:38.995533393 +0000 UTC m=+0.016518086 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:28:39.339 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:39 smithi016 podman[94656]: 2024-09-17 12:28:39.126428098 +0000 UTC m=+0.147412792 container create a3a9835cdcda138d7910bfffd425c115f40d5305a29a75bd0a7cb832052b3bb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-7498493, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-17T12:28:39.339 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:39 smithi016 ceph-mon[89784]: from='client.34203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:39.339 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:39 smithi016 ceph-mon[89784]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:39.339 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:39 smithi016 ceph-mon[89784]: from='client.44150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:39.388 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:28:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:28:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:28:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:28:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:28:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:28:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:28:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:28:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 7 2024-09-17T12:28:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:28:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:28:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:28:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:28:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:28:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 11, 2024-09-17T12:28:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 5 2024-09-17T12:28:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:28:39.390 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:28:39.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:39 smithi196 ceph-mon[80339]: from='client.34203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:39.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:39 smithi196 ceph-mon[80339]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:39.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:39 smithi196 ceph-mon[80339]: from='client.44150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:39.691 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:39 smithi016 podman[94656]: 2024-09-17 12:28:39.337556431 +0000 UTC m=+0.358541136 container init a3a9835cdcda138d7910bfffd425c115f40d5305a29a75bd0a7cb832052b3bb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-7498493, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:28:39.691 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:39 smithi016 podman[94656]: 2024-09-17 12:28:39.341858498 +0000 UTC m=+0.362843193 container start a3a9835cdcda138d7910bfffd425c115f40d5305a29a75bd0a7cb832052b3bb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-7498493, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-17T12:28:39.691 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:39 smithi016 podman[94656]: 2024-09-17 12:28:39.428995061 +0000 UTC m=+0.449979766 container attach a3a9835cdcda138d7910bfffd425c115f40d5305a29a75bd0a7cb832052b3bb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , RELEASE=main-7498493, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-09-17T12:28:39.691 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate[94692]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:28:39.691 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:39 smithi016 bash[94656]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:28:39.692 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate[94692]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:28:39.692 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:39 smithi016 bash[94656]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:28:39.692 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:28:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:28:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:28:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:28:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:28:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-09-17T12:28:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:28:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:28:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "5/23 daemons upgraded", 2024-09-17T12:28:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T12:28:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:28:39.693 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:28:40.040 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 osds down 2024-09-17T12:28:40.041 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-17T12:28:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: osd.0 (root=default,host=smithi016) is down 2024-09-17T12:28:40.243 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate[94692]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T12:28:40.244 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:39 smithi016 bash[94656]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T12:28:40.244 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate[94692]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T12:28:40.244 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:39 smithi016 bash[94656]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T12:28:40.244 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:40 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate[94692]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T12:28:40.244 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:40 smithi016 bash[94656]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T12:28:40.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:40 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/3388346342' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:40.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:40 smithi196 ceph-mon[80339]: pgmap v38: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 69/633 objects degraded (10.900%) 2024-09-17T12:28:40.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:40 smithi196 ceph-mon[80339]: from='client.44162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:40.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:40 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/1991784069' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:28:40.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:40 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/3388346342' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:40.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:40 smithi016 ceph-mon[87889]: pgmap v38: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 69/633 objects degraded (10.900%) 2024-09-17T12:28:40.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:40 smithi016 ceph-mon[87889]: from='client.44162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:40.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:40 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/1991784069' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:28:40.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:40 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/3388346342' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:40.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:40 smithi016 ceph-mon[89784]: pgmap v38: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 69/633 objects degraded (10.900%) 2024-09-17T12:28:40.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:40 smithi016 ceph-mon[89784]: from='client.44162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:28:40.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:40 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/1991784069' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:28:40.589 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:40 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate[94692]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T12:28:40.590 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:40 smithi016 bash[94656]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T12:28:40.590 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:40 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate[94692]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-17T12:28:40.590 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:40 smithi016 bash[94656]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-17T12:28:40.590 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:40 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate[94692]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T12:28:40.590 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:40 smithi016 bash[94656]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T12:28:40.590 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:40 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate[94692]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-17T12:28:40.590 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:40 smithi016 bash[94656]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-17T12:28:40.590 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:40 smithi016 podman[94656]: 2024-09-17 12:28:40.274424653 +0000 UTC m=+1.295409346 container died a3a9835cdcda138d7910bfffd425c115f40d5305a29a75bd0a7cb832052b3bb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, RELEASE=main-7498493, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-17T12:28:41.191 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:40 smithi016 podman[94870]: 2024-09-17 12:28:40.849618814 +0000 UTC m=+0.568613528 container remove a3a9835cdcda138d7910bfffd425c115f40d5305a29a75bd0a7cb832052b3bb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-activate, ceph=True, RELEASE=main-7498493, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-17T12:28:41.191 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:41 smithi016 podman[94931]: 2024-09-17 12:28:40.96429429 +0000 UTC m=+0.016428255 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:28:41.191 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:41 smithi016 podman[94931]: 2024-09-17 12:28:41.08394501 +0000 UTC m=+0.136078956 container create d12e6f26f048fa40ae36e598638f61fb01c68ec4d50c9ad7832d93c89e277c61 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=main-7498493, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-17T12:28:41.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:41 smithi196 ceph-mon[80339]: Health check failed: Degraded data redundancy: 69/633 objects degraded (10.900%), 21 pgs degraded (PG_DEGRADED) 2024-09-17T12:28:41.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:41 smithi016 ceph-mon[87889]: Health check failed: Degraded data redundancy: 69/633 objects degraded (10.900%), 21 pgs degraded (PG_DEGRADED) 2024-09-17T12:28:41.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:41 smithi016 ceph-mon[89784]: Health check failed: Degraded data redundancy: 69/633 objects degraded (10.900%), 21 pgs degraded (PG_DEGRADED) 2024-09-17T12:28:41.692 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:41 smithi016 podman[94931]: 2024-09-17 12:28:41.336382249 +0000 UTC m=+0.388516210 container init d12e6f26f048fa40ae36e598638f61fb01c68ec4d50c9ad7832d93c89e277c61 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, RELEASE=main-7498493, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-17T12:28:41.692 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:41 smithi016 podman[94931]: 2024-09-17 12:28:41.339922664 +0000 UTC m=+0.392056611 container start d12e6f26f048fa40ae36e598638f61fb01c68ec4d50c9ad7832d93c89e277c61 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-7498493) 2024-09-17T12:28:41.692 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:41 smithi016 ceph-osd[94951]: -- 172.21.15.16:0/265497758 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561a474c41a0 con 0x561a46697c00 2024-09-17T12:28:41.692 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:41 smithi016 ceph-osd[94951]: -- 172.21.15.16:0/265497758 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561a466b9a40 con 0x561a46697c00 2024-09-17T12:28:41.692 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:41 smithi016 bash[94931]: d12e6f26f048fa40ae36e598638f61fb01c68ec4d50c9ad7832d93c89e277c61 2024-09-17T12:28:41.692 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:41 smithi016 systemd[1]: Started Ceph osd.0 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:28:42.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:42 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:42.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:42 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:42.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:42 smithi196 ceph-mon[80339]: pgmap v39: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 69/633 objects degraded (10.900%) 2024-09-17T12:28:42.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:42 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:28:42.835 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:42 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:42.835 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:42 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:42.835 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:42 smithi016 ceph-mon[87889]: pgmap v39: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 69/633 objects degraded (10.900%) 2024-09-17T12:28:42.835 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:42 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:28:42.835 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:42 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:42.835 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:42 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:42.835 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:42 smithi016 ceph-mon[89784]: pgmap v39: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 69/633 objects degraded (10.900%) 2024-09-17T12:28:42.836 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:42 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:28:43.638 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:43 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0[94947]: 2024-09-17T12:28:43.220+0000 7fdc05ae9640 -1 Falling back to public interface 2024-09-17T12:28:44.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:44 smithi016 ceph-mon[87889]: pgmap v40: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 69/633 objects degraded (10.900%) 2024-09-17T12:28:44.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:44 smithi016 ceph-mon[89784]: pgmap v40: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 69/633 objects degraded (10.900%) 2024-09-17T12:28:45.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:44 smithi196 ceph-mon[80339]: pgmap v40: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 69/633 objects degraded (10.900%) 2024-09-17T12:28:46.342 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:28:46] "GET /metrics HTTP/1.1" 200 38502 "" "Prometheus/2.43.0" 2024-09-17T12:28:46.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:46.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:46.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-mon[87889]: pgmap v41: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 332 B/s rd, 0 op/s; 69/633 objects degraded (10.900%) 2024-09-17T12:28:46.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:46.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:46.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:46.659 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:46.659 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:46.659 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-mon[89784]: pgmap v41: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 332 B/s rd, 0 op/s; 69/633 objects degraded (10.900%) 2024-09-17T12:28:46.659 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:46.659 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:46.659 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:46.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:46 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:46 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:46 smithi196 ceph-mon[80339]: pgmap v41: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 332 B/s rd, 0 op/s; 69/633 objects degraded (10.900%) 2024-09-17T12:28:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:46 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:46 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:46 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:46.941 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0[94947]: 2024-09-17T12:28:46.783+0000 7fdc05ae9640 -1 osd.0 104 log_to_monitors true 2024-09-17T12:28:47.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:47 smithi016 ceph-mon[87889]: from='osd.0 [v2:172.21.15.16:6802/869620209,v1:172.21.15.16:6803/869620209]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-17T12:28:47.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:47 smithi016 ceph-mon[89784]: from='osd.0 [v2:172.21.15.16:6802/869620209,v1:172.21.15.16:6803/869620209]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-17T12:28:47.692 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:28:47 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0[94947]: 2024-09-17T12:28:47.484+0000 7fdbf988b640 -1 osd.0 104 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:28:47.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:47 smithi196 ceph-mon[80339]: from='osd.0 [v2:172.21.15.16:6802/869620209,v1:172.21.15.16:6803/869620209]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-17T12:28:48.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:48 smithi196 ceph-mon[80339]: from='osd.0 [v2:172.21.15.16:6802/869620209,v1:172.21.15.16:6803/869620209]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T12:28:48.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:48 smithi196 ceph-mon[80339]: osdmap e107: 8 total, 7 up, 8 in 2024-09-17T12:28:48.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:48 smithi196 ceph-mon[80339]: from='osd.0 [v2:172.21.15.16:6802/869620209,v1:172.21.15.16:6803/869620209]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:28:48.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:48 smithi196 ceph-mon[80339]: pgmap v43: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 69/633 objects degraded (10.900%) 2024-09-17T12:28:48.895 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:48 smithi016 ceph-mon[87889]: from='osd.0 [v2:172.21.15.16:6802/869620209,v1:172.21.15.16:6803/869620209]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T12:28:48.895 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:48 smithi016 ceph-mon[87889]: osdmap e107: 8 total, 7 up, 8 in 2024-09-17T12:28:48.895 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:48 smithi016 ceph-mon[87889]: from='osd.0 [v2:172.21.15.16:6802/869620209,v1:172.21.15.16:6803/869620209]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:28:48.895 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:48 smithi016 ceph-mon[87889]: pgmap v43: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 69/633 objects degraded (10.900%) 2024-09-17T12:28:48.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:48 smithi016 ceph-mon[89784]: from='osd.0 [v2:172.21.15.16:6802/869620209,v1:172.21.15.16:6803/869620209]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T12:28:48.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:48 smithi016 ceph-mon[89784]: osdmap e107: 8 total, 7 up, 8 in 2024-09-17T12:28:48.896 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:48 smithi016 ceph-mon[89784]: from='osd.0 [v2:172.21.15.16:6802/869620209,v1:172.21.15.16:6803/869620209]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:28:48.896 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:48 smithi016 ceph-mon[89784]: pgmap v43: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 69/633 objects degraded (10.900%) 2024-09-17T12:28:49.467 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:28:49.334+0000 7f9d37c73640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-17T12:28:49.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:28:49.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: osd.0 [v2:172.21.15.16:6802/869620209,v1:172.21.15.16:6803/869620209] boot 2024-09-17T12:28:49.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: osdmap e108: 8 total, 8 up, 8 in 2024-09-17T12:28:49.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T12:28:49.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:49.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:28:49.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:49.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:49.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:49.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:28:49.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:49.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:28:49.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:49.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:49.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:49.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-17T12:28:49.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:49 smithi196 ceph-mon[80339]: osdmap e109: 8 total, 8 up, 8 in 2024-09-17T12:28:49.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:28:49.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: osd.0 [v2:172.21.15.16:6802/869620209,v1:172.21.15.16:6803/869620209] boot 2024-09-17T12:28:49.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: osdmap e108: 8 total, 8 up, 8 in 2024-09-17T12:28:49.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T12:28:49.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:49.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:28:49.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:49.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:49.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:49.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:28:49.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:49.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:28:49.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:49.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:49.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:49.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-17T12:28:49.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[87889]: osdmap e109: 8 total, 8 up, 8 in 2024-09-17T12:28:49.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:28:49.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: osd.0 [v2:172.21.15.16:6802/869620209,v1:172.21.15.16:6803/869620209] boot 2024-09-17T12:28:49.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: osdmap e108: 8 total, 8 up, 8 in 2024-09-17T12:28:49.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T12:28:49.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:49.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:28:49.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:49.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:49.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:28:49.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:28:49.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:28:49.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:28:49.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:49.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:49.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:28:49.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-17T12:28:49.945 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:49 smithi016 ceph-mon[89784]: osdmap e109: 8 total, 8 up, 8 in 2024-09-17T12:28:50.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:50 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T12:28:50.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:50 smithi196 ceph-mon[80339]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-17T12:28:50.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:50 smithi196 ceph-mon[80339]: pgmap v46: 161 pgs: 2 active, 17 activating, 13 active+undersized, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 29/633 objects degraded (4.581%) 2024-09-17T12:28:50.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:50 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T12:28:50.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:50 smithi016 ceph-mon[87889]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-17T12:28:50.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:50 smithi016 ceph-mon[87889]: pgmap v46: 161 pgs: 2 active, 17 activating, 13 active+undersized, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 29/633 objects degraded (4.581%) 2024-09-17T12:28:50.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:50 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T12:28:50.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:50 smithi016 ceph-mon[89784]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-17T12:28:50.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:50 smithi016 ceph-mon[89784]: pgmap v46: 161 pgs: 2 active, 17 activating, 13 active+undersized, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 29/633 objects degraded (4.581%) 2024-09-17T12:28:51.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:51 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 29/633 objects degraded (4.581%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T12:28:51.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:51 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 29/633 objects degraded (4.581%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T12:28:51.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:51 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 29/633 objects degraded (4.581%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T12:28:52.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:52 smithi196 ceph-mon[80339]: pgmap v47: 161 pgs: 2 active, 17 activating, 13 active+undersized, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 29/633 objects degraded (4.581%) 2024-09-17T12:28:52.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:52 smithi016 ceph-mon[87889]: pgmap v47: 161 pgs: 2 active, 17 activating, 13 active+undersized, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 29/633 objects degraded (4.581%) 2024-09-17T12:28:52.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:52 smithi016 ceph-mon[89784]: pgmap v47: 161 pgs: 2 active, 17 activating, 13 active+undersized, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 29/633 objects degraded (4.581%) 2024-09-17T12:28:53.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:53 smithi016 ceph-mon[87889]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/633 objects degraded (4.581%), 8 pgs degraded) 2024-09-17T12:28:53.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:53 smithi016 ceph-mon[87889]: Cluster is now healthy 2024-09-17T12:28:53.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:53 smithi016 ceph-mon[89784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/633 objects degraded (4.581%), 8 pgs degraded) 2024-09-17T12:28:53.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:53 smithi016 ceph-mon[89784]: Cluster is now healthy 2024-09-17T12:28:54.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:53 smithi196 ceph-mon[80339]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/633 objects degraded (4.581%), 8 pgs degraded) 2024-09-17T12:28:54.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:53 smithi196 ceph-mon[80339]: Cluster is now healthy 2024-09-17T12:28:54.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:54 smithi016 ceph-mon[87889]: pgmap v48: 161 pgs: 2 active, 17 activating, 142 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-09-17T12:28:54.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:54 smithi016 ceph-mon[89784]: pgmap v48: 161 pgs: 2 active, 17 activating, 142 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-09-17T12:28:55.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:54 smithi196 ceph-mon[80339]: pgmap v48: 161 pgs: 2 active, 17 activating, 142 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-09-17T12:28:56.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:28:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:28:56] "GET /metrics HTTP/1.1" 200 38424 "" "Prometheus/2.43.0" 2024-09-17T12:28:56.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:56 smithi016 ceph-mon[87889]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:56.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:56 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:56.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:56 smithi016 ceph-mon[89784]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:56.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:56 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:57.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:56 smithi196 ceph-mon[80339]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:28:57.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:56 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:28:58.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:28:58 smithi016 ceph-mon[87889]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:28:58.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:28:58 smithi016 ceph-mon[89784]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:28:59.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:28:58 smithi196 ceph-mon[80339]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:29:00.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:00 smithi016 ceph-mon[87889]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T12:29:00.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:00 smithi016 ceph-mon[89784]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T12:29:01.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:00 smithi196 ceph-mon[80339]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T12:29:02.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:02 smithi016 ceph-mon[87889]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:02.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:02 smithi016 ceph-mon[89784]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:03.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:02 smithi196 ceph-mon[80339]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:04.997 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:04 smithi016 ceph-mon[87889]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:04.997 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:04 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:04.997 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:04 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:29:04.997 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:04 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:04.997 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:04 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-17T12:29:04.997 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:04 smithi016 ceph-mon[89784]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:04.998 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:04 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:04.998 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:04 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:29:04.998 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:04 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:04.998 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:04 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-17T12:29:05.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:04 smithi196 ceph-mon[80339]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:05.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:04 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:05.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:04 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:29:05.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:04 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:05.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:04 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-17T12:29:06.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:05 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T12:29:06.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:05 smithi016 ceph-mon[87889]: Upgrade: osd.1 is safe to restart 2024-09-17T12:29:06.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:05 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:06.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:05 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-17T12:29:06.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:05 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:29:06.063 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:05 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T12:29:06.063 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:05 smithi016 ceph-mon[89784]: Upgrade: osd.1 is safe to restart 2024-09-17T12:29:06.064 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:05 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:06.064 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:05 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-17T12:29:06.064 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:05 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:29:06.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:05 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T12:29:06.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:05 smithi196 ceph-mon[80339]: Upgrade: osd.1 is safe to restart 2024-09-17T12:29:06.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:05 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:06.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:05 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-17T12:29:06.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:05 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:29:06.395 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:29:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:29:06] "GET /metrics HTTP/1.1" 200 38566 "" "Prometheus/2.43.0" 2024-09-17T12:29:07.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:06 smithi016 ceph-mon[89784]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:07.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:06 smithi016 ceph-mon[89784]: Upgrade: Updating osd.1 2024-09-17T12:29:07.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:06 smithi016 ceph-mon[89784]: Deploying daemon osd.1 on smithi016 2024-09-17T12:29:07.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:06 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:07.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:06 smithi016 ceph-mon[87889]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:07.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:06 smithi016 ceph-mon[87889]: Upgrade: Updating osd.1 2024-09-17T12:29:07.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:06 smithi016 ceph-mon[87889]: Deploying daemon osd.1 on smithi016 2024-09-17T12:29:07.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:06 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:07.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:06 smithi196 ceph-mon[80339]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:07.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:06 smithi196 ceph-mon[80339]: Upgrade: Updating osd.1 2024-09-17T12:29:07.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:06 smithi196 ceph-mon[80339]: Deploying daemon osd.1 on smithi016 2024-09-17T12:29:07.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:06 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:09.072 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:08 smithi016 ceph-mon[87889]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:09.072 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:08 smithi016 systemd[1]: Stopping Ceph osd.1 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:29:09.072 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:08 smithi016 ceph-mon[89784]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:09.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:08 smithi196 ceph-mon[80339]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:09.441 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:09 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1[44110]: 2024-09-17T12:29:09.101+0000 7f560a86a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:29:09.441 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:09 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1[44110]: 2024-09-17T12:29:09.101+0000 7f560a86a640 -1 osd.1 109 *** Got signal Terminated *** 2024-09-17T12:29:09.441 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:09 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1[44110]: 2024-09-17T12:29:09.101+0000 7f560a86a640 -1 osd.1 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:29:10.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:09 smithi016 ceph-mon[87889]: osd.1 marked itself down and dead 2024-09-17T12:29:10.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:09 smithi016 ceph-mon[89784]: osd.1 marked itself down and dead 2024-09-17T12:29:10.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:09 smithi196 ceph-mon[80339]: osd.1 marked itself down and dead 2024-09-17T12:29:10.358 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:29:10.578 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:10 smithi016 podman[99102]: 2024-09-17 12:29:10.301616944 +0000 UTC m=+1.310191258 container died 3ad94c30c9c9c82dd182f3f51444e198b7fc9cd03c780b8abbc2905aefa193db (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=reef-19e2a2b, ceph=True) 2024-09-17T12:29:10.912 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:10 smithi016 podman[99102]: 2024-09-17 12:29:10.676037637 +0000 UTC m=+1.684611950 container cleanup 3ad94c30c9c9c82dd182f3f51444e198b7fc9cd03c780b8abbc2905aefa193db (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b) 2024-09-17T12:29:10.912 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:10 smithi016 bash[99102]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1 2024-09-17T12:29:10.983 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:29:10.983 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (7m) 25s ago 8m 25.3M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:29:10.984 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (7m) 66s ago 7m 89.2M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:29:10.984 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (2m) 25s ago 6m 57.5M - 3.8 980a0bfcd684 843db1407344 2024-09-17T12:29:10.984 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (117s) 66s ago 11m 453M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:29:10.984 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (2m) 25s ago 13m 513M - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:29:10.984 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (104s) 25s ago 13m 44.2M 2048M 19.3.0-4937-g74984932 6c1fcab85602 4a83b787c605 2024-09-17T12:29:10.984 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (71s) 66s ago 12m 24.7M 2048M 19.3.0-4937-g74984932 6c1fcab85602 dc48dfc9e40c 2024-09-17T12:29:10.984 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (84s) 25s ago 12m 39.1M 2048M 19.3.0-4937-g74984932 6c1fcab85602 aa5074acd22a 2024-09-17T12:29:10.984 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (7m) 25s ago 7m 15.6M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:29:10.984 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (7m) 66s ago 7m 15.4M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:29:10.984 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (30s) 25s ago 11m 11.5M 2737M 19.3.0-4937-g74984932 6c1fcab85602 d12e6f26f048 2024-09-17T12:29:10.985 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (11m) 25s ago 11m 55.0M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 3ad94c30c9c9 2024-09-17T12:29:10.985 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (10m) 25s ago 10m 56.1M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 1fa310695876 2024-09-17T12:29:10.985 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (10m) 25s ago 10m 61.5M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 4a4f8eeaff13 2024-09-17T12:29:10.985 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (10m) 66s ago 10m 55.7M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 afa20056b398 2024-09-17T12:29:10.985 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (9m) 66s ago 9m 51.2M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 f0dcabb8902f 2024-09-17T12:29:10.985 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (9m) 66s ago 9m 60.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:29:10.985 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (8m) 66s ago 8m 61.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:29:10.985 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (2m) 66s ago 8m 42.7M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:29:10.985 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (6m) 25s ago 6m 80.0M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:29:10.985 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (6m) 66s ago 6m 80.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:29:10.985 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (6m) 25s ago 6m 82.2M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:29:10.985 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (6m) 66s ago 6m 79.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:29:11.191 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:10 smithi016 podman[99150]: 2024-09-17 12:29:10.910667583 +0000 UTC m=+0.603522891 container remove 3ad94c30c9c9c82dd182f3f51444e198b7fc9cd03c780b8abbc2905aefa193db (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-09-17T12:29:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:10 smithi016 ceph-mon[87889]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:10 smithi016 ceph-mon[87889]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:29:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:10 smithi016 ceph-mon[87889]: osdmap e110: 8 total, 7 up, 8 in 2024-09-17T12:29:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:10 smithi016 ceph-mon[89784]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:10 smithi016 ceph-mon[89784]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:29:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:10 smithi016 ceph-mon[89784]: osdmap e110: 8 total, 7 up, 8 in 2024-09-17T12:29:11.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:10 smithi196 ceph-mon[80339]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:11.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:10 smithi196 ceph-mon[80339]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:29:11.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:10 smithi196 ceph-mon[80339]: osdmap e110: 8 total, 7 up, 8 in 2024-09-17T12:29:11.373 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:29:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:29:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:29:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:29:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:29:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:29:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:29:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:29:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 6, 2024-09-17T12:29:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 1 2024-09-17T12:29:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:29:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:29:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:29:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:29:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:29:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 10, 2024-09-17T12:29:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 6 2024-09-17T12:29:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:29:11.375 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:29:11.615 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:11 smithi016 podman[99306]: 2024-09-17 12:29:11.295229813 +0000 UTC m=+0.109249096 container create 59237825065ba11ed3fb16475faf3e5584c4f69e45974f495771660c25ab1678 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-7498493, org.label-schema.schema-version=1.0) 2024-09-17T12:29:11.615 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:11 smithi016 podman[99306]: 2024-09-17 12:29:11.203370402 +0000 UTC m=+0.017389687 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:29:11.615 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:11 smithi016 podman[99306]: 2024-09-17 12:29:11.522799068 +0000 UTC m=+0.336818359 container init 59237825065ba11ed3fb16475faf3e5584c4f69e45974f495771660c25ab1678 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=main-7498493, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731) 2024-09-17T12:29:11.616 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:11 smithi016 podman[99306]: 2024-09-17 12:29:11.526398676 +0000 UTC m=+0.340417970 container start 59237825065ba11ed3fb16475faf3e5584c4f69e45974f495771660c25ab1678 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=main-7498493, org.label-schema.build-date=20240731) 2024-09-17T12:29:11.616 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:11 smithi016 podman[99306]: 2024-09-17 12:29:11.614453168 +0000 UTC m=+0.428472457 container attach 59237825065ba11ed3fb16475faf3e5584c4f69e45974f495771660c25ab1678 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-7498493, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-17T12:29:11.678 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:29:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:29:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:29:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:29:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:29:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-09-17T12:29:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:29:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:29:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "6/23 daemons upgraded", 2024-09-17T12:29:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T12:29:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:29:11.679 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:29:11.941 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:11 smithi016 podman[99306]: 2024-09-17 12:29:11.732020396 +0000 UTC m=+0.546039681 container died 59237825065ba11ed3fb16475faf3e5584c4f69e45974f495771660c25ab1678 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=main-7498493, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-17T12:29:12.039 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 osds down 2024-09-17T12:29:12.039 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-17T12:29:12.039 INFO:teuthology.orchestra.run.smithi016.stdout: osd.1 (root=default,host=smithi016) is down 2024-09-17T12:29:12.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:11 smithi196 ceph-mon[80339]: from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:11 smithi196 ceph-mon[80339]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:11 smithi196 ceph-mon[80339]: osdmap e111: 8 total, 7 up, 8 in 2024-09-17T12:29:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:11 smithi196 ceph-mon[80339]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:11 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/3056539405' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:12.336 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:11 smithi016 ceph-mon[87889]: from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:12.336 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:11 smithi016 ceph-mon[87889]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:12.336 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:11 smithi016 ceph-mon[87889]: osdmap e111: 8 total, 7 up, 8 in 2024-09-17T12:29:12.336 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:11 smithi016 ceph-mon[87889]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:12.336 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:11 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/3056539405' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:12.336 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:11 smithi016 ceph-mon[89784]: from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:12.337 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:11 smithi016 ceph-mon[89784]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:12.337 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:11 smithi016 ceph-mon[89784]: osdmap e111: 8 total, 7 up, 8 in 2024-09-17T12:29:12.337 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:11 smithi016 ceph-mon[89784]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:12.337 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:11 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/3056539405' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:12.691 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:12 smithi016 podman[99354]: 2024-09-17 12:29:12.335089642 +0000 UTC m=+0.593648519 container remove 59237825065ba11ed3fb16475faf3e5584c4f69e45974f495771660c25ab1678 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-7498493, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-17T12:29:12.691 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:12 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.1.service: Deactivated successfully. 2024-09-17T12:29:12.691 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:12 smithi016 systemd[1]: Stopped Ceph osd.1 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:29:12.691 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:12 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.1.service: Consumed 7.692s CPU time. 2024-09-17T12:29:12.691 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:12 smithi016 systemd[1]: Starting Ceph osd.1 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:29:13.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:12 smithi016 ceph-mon[87889]: pgmap v59: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:13.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:12 smithi016 ceph-mon[87889]: from='client.34269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:13.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:12 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/2888888993' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:29:13.082 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:12 smithi016 ceph-mon[89784]: pgmap v59: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:13.082 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:12 smithi016 ceph-mon[89784]: from='client.34269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:13.082 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:12 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/2888888993' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:29:13.082 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:12 smithi016 podman[99474]: 2024-09-17 12:29:12.76995959 +0000 UTC m=+0.101572580 container create 1e1f519aa12623e34107b070d78b7836f36775ff1011e51935a2761903abad75 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate, ceph=True, RELEASE=main-7498493, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:29:13.082 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:12 smithi016 podman[99474]: 2024-09-17 12:29:12.684999708 +0000 UTC m=+0.016612672 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:29:13.082 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:12 smithi016 podman[99474]: 2024-09-17 12:29:12.989076637 +0000 UTC m=+0.320689585 container init 1e1f519aa12623e34107b070d78b7836f36775ff1011e51935a2761903abad75 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-7498493, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:29:13.082 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:12 smithi016 podman[99474]: 2024-09-17 12:29:12.992552928 +0000 UTC m=+0.324165874 container start 1e1f519aa12623e34107b070d78b7836f36775ff1011e51935a2761903abad75 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=main-7498493, GIT_BRANCH=HEAD) 2024-09-17T12:29:13.083 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 podman[99474]: 2024-09-17 12:29:13.080664109 +0000 UTC m=+0.412277065 container attach 1e1f519aa12623e34107b070d78b7836f36775ff1011e51935a2761903abad75 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-17T12:29:13.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:12 smithi196 ceph-mon[80339]: pgmap v59: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:13.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:12 smithi196 ceph-mon[80339]: from='client.34269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:13.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:12 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/2888888993' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:29:13.441 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate[99489]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:29:13.441 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 bash[99474]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:29:13.441 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate[99489]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:29:13.441 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 bash[99474]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:29:13.892 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate[99489]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T12:29:13.892 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 bash[99474]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T12:29:13.892 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate[99489]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T12:29:13.892 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 bash[99474]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T12:29:13.892 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate[99489]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T12:29:13.892 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 bash[99474]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T12:29:14.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:14 smithi016 ceph-mon[87889]: Health check failed: Degraded data redundancy: 8/633 objects degraded (1.264%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:14.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:14 smithi016 ceph-mon[89784]: Health check failed: Degraded data redundancy: 8/633 objects degraded (1.264%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:14.192 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate[99489]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T12:29:14.192 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 bash[99474]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T12:29:14.192 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate[99489]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-17T12:29:14.192 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 bash[99474]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-17T12:29:14.192 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate[99489]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T12:29:14.192 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 bash[99474]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T12:29:14.192 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate[99489]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-17T12:29:14.192 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 bash[99474]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-17T12:29:14.192 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:13 smithi016 podman[99474]: 2024-09-17 12:29:13.920526999 +0000 UTC m=+1.252139957 container died 1e1f519aa12623e34107b070d78b7836f36775ff1011e51935a2761903abad75 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=main-7498493, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-17T12:29:14.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:14 smithi196 ceph-mon[80339]: Health check failed: Degraded data redundancy: 8/633 objects degraded (1.264%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:14.785 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:14 smithi016 podman[99618]: 2024-09-17 12:29:14.484273704 +0000 UTC m=+0.557199676 container remove 1e1f519aa12623e34107b070d78b7836f36775ff1011e51935a2761903abad75 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, RELEASE=main-7498493, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:29:14.785 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:14 smithi016 podman[99677]: 2024-09-17 12:29:14.685324376 +0000 UTC m=+0.101026745 container create cd2f6e2964dd8a6489d4313a1176ed6e69f77cc1871bfd4eafe63fdc4f0d224b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-7498493, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:29:14.785 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:14 smithi016 podman[99677]: 2024-09-17 12:29:14.60124643 +0000 UTC m=+0.016948764 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:29:15.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:15 smithi016 ceph-mon[87889]: pgmap v60: 161 pgs: 5 active+undersized, 19 stale+active+clean, 2 active+undersized+degraded, 135 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 8/633 objects degraded (1.264%) 2024-09-17T12:29:15.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:15 smithi016 ceph-mon[89784]: pgmap v60: 161 pgs: 5 active+undersized, 19 stale+active+clean, 2 active+undersized+degraded, 135 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 8/633 objects degraded (1.264%) 2024-09-17T12:29:15.191 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:14 smithi016 podman[99677]: 2024-09-17 12:29:14.879423191 +0000 UTC m=+0.295125510 container init cd2f6e2964dd8a6489d4313a1176ed6e69f77cc1871bfd4eafe63fdc4f0d224b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=main-7498493, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-17T12:29:15.191 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:14 smithi016 podman[99677]: 2024-09-17 12:29:14.882710017 +0000 UTC m=+0.298412326 container start cd2f6e2964dd8a6489d4313a1176ed6e69f77cc1871bfd4eafe63fdc4f0d224b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-7498493, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:29:15.192 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:14 smithi016 ceph-osd[99696]: -- 172.21.15.16:0/1938837829 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563814825a40 con 0x5638155f4400 2024-09-17T12:29:15.192 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:14 smithi016 bash[99677]: cd2f6e2964dd8a6489d4313a1176ed6e69f77cc1871bfd4eafe63fdc4f0d224b 2024-09-17T12:29:15.192 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:14 smithi016 systemd[1]: Started Ceph osd.1 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:29:15.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:15 smithi196 ceph-mon[80339]: pgmap v60: 161 pgs: 5 active+undersized, 19 stale+active+clean, 2 active+undersized+degraded, 135 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 8/633 objects degraded (1.264%) 2024-09-17T12:29:16.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:16 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:16.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:16 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:16.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:16 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:29:16.362 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1[99692]: 2024-09-17T12:29:16.281+0000 7f5fc0389640 -1 Falling back to public interface 2024-09-17T12:29:16.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:16 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:16.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:16 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:16.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:16 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:29:16.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:16 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:16.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:16 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:16.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:16 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:29:16.363 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:29:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:29:16] "GET /metrics HTTP/1.1" 200 38566 "" "Prometheus/2.43.0" 2024-09-17T12:29:17.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:17 smithi016 ceph-mon[87889]: pgmap v61: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 87/633 objects degraded (13.744%) 2024-09-17T12:29:17.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:17 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:17.324 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:17 smithi016 ceph-mon[89784]: pgmap v61: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 87/633 objects degraded (13.744%) 2024-09-17T12:29:17.324 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:17 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:17.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:17 smithi196 ceph-mon[80339]: pgmap v61: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 87/633 objects degraded (13.744%) 2024-09-17T12:29:17.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:17 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:19.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:19 smithi016 ceph-mon[87889]: pgmap v62: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 87/633 objects degraded (13.744%) 2024-09-17T12:29:19.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:19 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:19.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:19 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:29:19.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:19 smithi016 ceph-mon[89784]: pgmap v62: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 87/633 objects degraded (13.744%) 2024-09-17T12:29:19.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:19 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:19.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:19 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:29:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:19 smithi196 ceph-mon[80339]: pgmap v62: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 87/633 objects degraded (13.744%) 2024-09-17T12:29:19.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:19 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:19.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:19 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:29:19.941 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:19 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1[99692]: 2024-09-17T12:29:19.586+0000 7f5fc0389640 -1 osd.1 109 log_to_monitors true 2024-09-17T12:29:20.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:20 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:20.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:20 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:20.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:20 smithi016 ceph-mon[87889]: from='osd.1 [v2:172.21.15.16:6810/2525927145,v1:172.21.15.16:6811/2525927145]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-17T12:29:20.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:20 smithi016 ceph-mon[87889]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-17T12:29:20.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:20 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:20.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:20 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:20.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:20 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:20.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:20 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:20.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:20 smithi016 ceph-mon[89784]: from='osd.1 [v2:172.21.15.16:6810/2525927145,v1:172.21.15.16:6811/2525927145]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-17T12:29:20.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:20 smithi016 ceph-mon[89784]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-17T12:29:20.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:20 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:20.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:20 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:20.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:20 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:20.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:20 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:20.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:20 smithi196 ceph-mon[80339]: from='osd.1 [v2:172.21.15.16:6810/2525927145,v1:172.21.15.16:6811/2525927145]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-17T12:29:20.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:20 smithi196 ceph-mon[80339]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-17T12:29:20.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:20 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:20.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:20 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:20.941 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:29:20 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1[99692]: 2024-09-17T12:29:20.521+0000 7f5fb412b640 -1 osd.1 109 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:29:21.385 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:21 smithi016 ceph-mon[87889]: pgmap v63: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 87/633 objects degraded (13.744%) 2024-09-17T12:29:21.385 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:21 smithi016 ceph-mon[87889]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T12:29:21.385 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:21 smithi016 ceph-mon[87889]: osdmap e112: 8 total, 7 up, 8 in 2024-09-17T12:29:21.385 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:21 smithi016 ceph-mon[87889]: from='osd.1 [v2:172.21.15.16:6810/2525927145,v1:172.21.15.16:6811/2525927145]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:29:21.385 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:21 smithi016 ceph-mon[87889]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:29:21.385 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:21 smithi016 ceph-mon[87889]: from='osd.1 ' entity='osd.1' 2024-09-17T12:29:21.385 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:21 smithi016 ceph-mon[89784]: pgmap v63: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 87/633 objects degraded (13.744%) 2024-09-17T12:29:21.385 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:21 smithi016 ceph-mon[89784]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T12:29:21.386 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:21 smithi016 ceph-mon[89784]: osdmap e112: 8 total, 7 up, 8 in 2024-09-17T12:29:21.386 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:21 smithi016 ceph-mon[89784]: from='osd.1 [v2:172.21.15.16:6810/2525927145,v1:172.21.15.16:6811/2525927145]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:29:21.386 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:21 smithi016 ceph-mon[89784]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:29:21.386 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:21 smithi016 ceph-mon[89784]: from='osd.1 ' entity='osd.1' 2024-09-17T12:29:21.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:21 smithi196 ceph-mon[80339]: pgmap v63: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 87/633 objects degraded (13.744%) 2024-09-17T12:29:21.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:21 smithi196 ceph-mon[80339]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T12:29:21.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:21 smithi196 ceph-mon[80339]: osdmap e112: 8 total, 7 up, 8 in 2024-09-17T12:29:21.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:21 smithi196 ceph-mon[80339]: from='osd.1 [v2:172.21.15.16:6810/2525927145,v1:172.21.15.16:6811/2525927145]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:29:21.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:21 smithi196 ceph-mon[80339]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:29:21.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:21 smithi196 ceph-mon[80339]: from='osd.1 ' entity='osd.1' 2024-09-17T12:29:22.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:22 smithi016 ceph-mon[87889]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:29:22.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:22 smithi016 ceph-mon[87889]: osd.1 [v2:172.21.15.16:6810/2525927145,v1:172.21.15.16:6811/2525927145] boot 2024-09-17T12:29:22.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:22 smithi016 ceph-mon[87889]: osdmap e113: 8 total, 8 up, 8 in 2024-09-17T12:29:22.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:22 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T12:29:22.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:22 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 87/633 objects degraded (13.744%), 23 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:22 smithi016 ceph-mon[89784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:29:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:22 smithi016 ceph-mon[89784]: osd.1 [v2:172.21.15.16:6810/2525927145,v1:172.21.15.16:6811/2525927145] boot 2024-09-17T12:29:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:22 smithi016 ceph-mon[89784]: osdmap e113: 8 total, 8 up, 8 in 2024-09-17T12:29:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:22 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T12:29:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:22 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 87/633 objects degraded (13.744%), 23 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:22.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:22 smithi196 ceph-mon[80339]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:29:22.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:22 smithi196 ceph-mon[80339]: osd.1 [v2:172.21.15.16:6810/2525927145,v1:172.21.15.16:6811/2525927145] boot 2024-09-17T12:29:22.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:22 smithi196 ceph-mon[80339]: osdmap e113: 8 total, 8 up, 8 in 2024-09-17T12:29:22.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:22 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T12:29:22.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:22 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 87/633 objects degraded (13.744%), 23 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:23.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:29:22 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:29:22.928+0000 7f9d37c73640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-17T12:29:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:23 smithi196 ceph-mon[80339]: pgmap v65: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 87/633 objects degraded (13.744%) 2024-09-17T12:29:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:23 smithi196 ceph-mon[80339]: osdmap e114: 8 total, 8 up, 8 in 2024-09-17T12:29:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:23 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:23 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:23 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:29:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:23 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:29:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:23 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:23 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:29:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:23 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:23 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:23 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:23.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:23 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-17T12:29:23.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[87889]: pgmap v65: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 87/633 objects degraded (13.744%) 2024-09-17T12:29:23.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[87889]: osdmap e114: 8 total, 8 up, 8 in 2024-09-17T12:29:23.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:23.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:23.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:29:23.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:29:23.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:23.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:29:23.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:23.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:23.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:23.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-17T12:29:23.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[89784]: pgmap v65: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 87/633 objects degraded (13.744%) 2024-09-17T12:29:23.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[89784]: osdmap e114: 8 total, 8 up, 8 in 2024-09-17T12:29:23.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:23.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:23.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:29:23.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:29:23.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:23.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:29:23.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:23.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:23.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:23.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:23 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-17T12:29:24.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:24 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T12:29:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:24 smithi196 ceph-mon[80339]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-17T12:29:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:24 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:24.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:24 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T12:29:24.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:24 smithi016 ceph-mon[87889]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-17T12:29:24.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:24 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:24.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:24 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T12:29:24.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:24 smithi016 ceph-mon[89784]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-17T12:29:24.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:24 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:25.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:25 smithi196 ceph-mon[80339]: pgmap v68: 161 pgs: 6 peering, 35 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 81/633 objects degraded (12.796%) 2024-09-17T12:29:25.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:25 smithi016 ceph-mon[87889]: pgmap v68: 161 pgs: 6 peering, 35 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 81/633 objects degraded (12.796%) 2024-09-17T12:29:25.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:25 smithi016 ceph-mon[89784]: pgmap v68: 161 pgs: 6 peering, 35 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 81/633 objects degraded (12.796%) 2024-09-17T12:29:26.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:26 smithi016 ceph-mon[87889]: pgmap v69: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:26.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:26 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:26.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:26 smithi016 ceph-mon[87889]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/633 objects degraded (12.796%), 22 pgs degraded) 2024-09-17T12:29:26.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:26 smithi016 ceph-mon[87889]: Cluster is now healthy 2024-09-17T12:29:26.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:29:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:29:26] "GET /metrics HTTP/1.1" 200 38633 "" "Prometheus/2.43.0" 2024-09-17T12:29:26.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:26 smithi016 ceph-mon[89784]: pgmap v69: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:26.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:26 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:26.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:26 smithi016 ceph-mon[89784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/633 objects degraded (12.796%), 22 pgs degraded) 2024-09-17T12:29:26.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:26 smithi016 ceph-mon[89784]: Cluster is now healthy 2024-09-17T12:29:26.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:26 smithi196 ceph-mon[80339]: pgmap v69: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:26.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:26 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:26.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:26 smithi196 ceph-mon[80339]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/633 objects degraded (12.796%), 22 pgs degraded) 2024-09-17T12:29:26.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:26 smithi196 ceph-mon[80339]: Cluster is now healthy 2024-09-17T12:29:28.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:28 smithi016 ceph-mon[87889]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 0 op/s 2024-09-17T12:29:28.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:28 smithi016 ceph-mon[89784]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 0 op/s 2024-09-17T12:29:29.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:28 smithi196 ceph-mon[80339]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 0 op/s 2024-09-17T12:29:30.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:30 smithi016 ceph-mon[87889]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:30.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:30 smithi016 ceph-mon[89784]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:31.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:30 smithi196 ceph-mon[80339]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:32.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:32 smithi016 ceph-mon[87889]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T12:29:32.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:32 smithi016 ceph-mon[89784]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T12:29:33.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:32 smithi196 ceph-mon[80339]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T12:29:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:34 smithi016 ceph-mon[87889]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-09-17T12:29:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:34 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:29:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:34 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:34 smithi016 ceph-mon[89784]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-09-17T12:29:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:34 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:34 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:29:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:34 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:34 smithi196 ceph-mon[80339]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-09-17T12:29:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:34 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:34 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:29:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:34 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:36.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:29:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:29:36] "GET /metrics HTTP/1.1" 200 38557 "" "Prometheus/2.43.0" 2024-09-17T12:29:37.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:36 smithi016 ceph-mon[87889]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:37.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:36 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:36 smithi016 ceph-mon[89784]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:36 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:37.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:36 smithi196 ceph-mon[80339]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:37.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:36 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:39.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:38 smithi016 ceph-mon[87889]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:39.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:38 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-17T12:29:39.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:38 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T12:29:39.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:38 smithi016 ceph-mon[87889]: Upgrade: osd.2 is safe to restart 2024-09-17T12:29:39.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:38 smithi016 ceph-mon[89784]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:39.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:38 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-17T12:29:39.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:38 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T12:29:39.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:38 smithi016 ceph-mon[89784]: Upgrade: osd.2 is safe to restart 2024-09-17T12:29:39.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:38 smithi196 ceph-mon[80339]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:39.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:38 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-17T12:29:39.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:38 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T12:29:39.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:38 smithi196 ceph-mon[80339]: Upgrade: osd.2 is safe to restart 2024-09-17T12:29:40.559 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:40 smithi016 ceph-mon[87889]: Upgrade: Updating osd.2 2024-09-17T12:29:40.559 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:40 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:40.559 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:40 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-17T12:29:40.559 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:40 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:29:40.559 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:40 smithi016 ceph-mon[87889]: Deploying daemon osd.2 on smithi016 2024-09-17T12:29:40.559 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:40 smithi016 ceph-mon[87889]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:40.560 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:40 smithi016 ceph-mon[89784]: Upgrade: Updating osd.2 2024-09-17T12:29:40.560 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:40 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:40.560 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:40 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-17T12:29:40.560 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:40 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:29:40.560 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:40 smithi016 ceph-mon[89784]: Deploying daemon osd.2 on smithi016 2024-09-17T12:29:40.560 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:40 smithi016 ceph-mon[89784]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:40.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:40 smithi196 ceph-mon[80339]: Upgrade: Updating osd.2 2024-09-17T12:29:40.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:40 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:40.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:40 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-17T12:29:40.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:40 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:29:40.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:40 smithi196 ceph-mon[80339]: Deploying daemon osd.2 on smithi016 2024-09-17T12:29:40.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:40 smithi196 ceph-mon[80339]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:29:42.398 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:29:42.513 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:42 smithi016 systemd[1]: Stopping Ceph osd.2 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:29:42.889 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:42 smithi016 ceph-mon[87889]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:42.889 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:42 smithi016 ceph-mon[87889]: osd.2 marked itself down and dead 2024-09-17T12:29:42.890 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:42 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2[50096]: 2024-09-17T12:29:42.510+0000 7fc11705e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:29:42.890 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:42 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2[50096]: 2024-09-17T12:29:42.510+0000 7fc11705e640 -1 osd.2 114 *** Got signal Terminated *** 2024-09-17T12:29:42.890 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:42 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2[50096]: 2024-09-17T12:29:42.510+0000 7fc11705e640 -1 osd.2 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:29:42.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:42 smithi016 ceph-mon[89784]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:42.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:42 smithi016 ceph-mon[89784]: osd.2 marked itself down and dead 2024-09-17T12:29:43.013 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:29:43.013 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (7m) 23s ago 8m 25.0M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:29:43.014 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (8m) 98s ago 8m 89.2M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:29:43.014 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (2m) 23s ago 7m 57.5M - 3.8 980a0bfcd684 843db1407344 2024-09-17T12:29:43.014 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (2m) 98s ago 12m 453M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:29:43.014 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (3m) 23s ago 14m 516M - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:29:43.014 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (2m) 23s ago 14m 45.7M 2048M 19.3.0-4937-g74984932 6c1fcab85602 4a83b787c605 2024-09-17T12:29:43.014 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (103s) 98s ago 12m 24.7M 2048M 19.3.0-4937-g74984932 6c1fcab85602 dc48dfc9e40c 2024-09-17T12:29:43.014 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (116s) 23s ago 12m 40.3M 2048M 19.3.0-4937-g74984932 6c1fcab85602 aa5074acd22a 2024-09-17T12:29:43.014 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (8m) 23s ago 8m 16.2M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:29:43.014 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (8m) 98s ago 8m 15.4M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:29:43.014 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (62s) 23s ago 12m 43.0M 2737M 19.3.0-4937-g74984932 6c1fcab85602 d12e6f26f048 2024-09-17T12:29:43.014 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (28s) 23s ago 11m 14.6M 2737M 19.3.0-4937-g74984932 6c1fcab85602 cd2f6e2964dd 2024-09-17T12:29:43.014 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (11m) 23s ago 11m 57.5M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 1fa310695876 2024-09-17T12:29:43.014 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (11m) 23s ago 11m 62.1M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 4a4f8eeaff13 2024-09-17T12:29:43.015 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (10m) 98s ago 10m 55.7M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 afa20056b398 2024-09-17T12:29:43.015 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (10m) 98s ago 10m 51.2M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 f0dcabb8902f 2024-09-17T12:29:43.015 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (9m) 98s ago 9m 60.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:29:43.015 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (9m) 98s ago 9m 61.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:29:43.015 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (2m) 98s ago 8m 42.7M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:29:43.015 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (7m) 23s ago 7m 80.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:29:43.015 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (7m) 98s ago 7m 80.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:29:43.015 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (7m) 23s ago 7m 82.5M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:29:43.015 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (7m) 98s ago 7m 79.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:29:43.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:42 smithi196 ceph-mon[80339]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:29:43.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:42 smithi196 ceph-mon[80339]: osd.2 marked itself down and dead 2024-09-17T12:29:43.191 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:42 smithi016 podman[104146]: 2024-09-17 12:29:42.888108907 +0000 UTC m=+0.482946948 container died 1fa310695876f15b06ac9d8b6f0ff97b6054cd43170d020de9afb21226c25e0a (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T12:29:43.367 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:29:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:29:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:29:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:29:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:29:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:29:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:29:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:29:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 5, 2024-09-17T12:29:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:29:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:29:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:29:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:29:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:29:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:29:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 9, 2024-09-17T12:29:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 7 2024-09-17T12:29:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:29:43.369 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:29:43.685 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:29:43.689 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:29:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:29:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:29:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:29:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-09-17T12:29:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:29:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:29:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "7/23 daemons upgraded", 2024-09-17T12:29:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T12:29:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:29:43.690 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:29:43.691 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:43 smithi016 podman[104146]: 2024-09-17 12:29:43.428643168 +0000 UTC m=+1.023481201 container cleanup 1fa310695876f15b06ac9d8b6f0ff97b6054cd43170d020de9afb21226c25e0a (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:29:43.691 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:43 smithi016 bash[104146]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2 2024-09-17T12:29:43.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:43 smithi016 ceph-mon[89784]: from='client.44222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:43.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:43 smithi016 ceph-mon[89784]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:29:43.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:43 smithi016 ceph-mon[89784]: osdmap e115: 8 total, 7 up, 8 in 2024-09-17T12:29:43.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:43 smithi016 ceph-mon[89784]: from='client.44228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:43.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:43 smithi016 ceph-mon[89784]: from='client.44234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:43.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:43 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/2674252866' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:43.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:43 smithi016 ceph-mon[87889]: from='client.44222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:43.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:43 smithi016 ceph-mon[87889]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:29:43.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:43 smithi016 ceph-mon[87889]: osdmap e115: 8 total, 7 up, 8 in 2024-09-17T12:29:43.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:43 smithi016 ceph-mon[87889]: from='client.44228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:43.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:43 smithi016 ceph-mon[87889]: from='client.44234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:43.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:43 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/2674252866' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:43.941 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:43 smithi016 podman[104203]: 2024-09-17 12:29:43.747225443 +0000 UTC m=+0.861594716 container remove 1fa310695876f15b06ac9d8b6f0ff97b6054cd43170d020de9afb21226c25e0a (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:29:44.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:43 smithi196 ceph-mon[80339]: from='client.44222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:44.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:43 smithi196 ceph-mon[80339]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:29:44.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:43 smithi196 ceph-mon[80339]: osdmap e115: 8 total, 7 up, 8 in 2024-09-17T12:29:44.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:43 smithi196 ceph-mon[80339]: from='client.44228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:44.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:43 smithi196 ceph-mon[80339]: from='client.44234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:44.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:43 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/2674252866' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:44.060 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 osds down 2024-09-17T12:29:44.060 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-17T12:29:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: osd.2 (root=default,host=smithi016) is down 2024-09-17T12:29:44.343 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:44 smithi016 podman[104362]: 2024-09-17 12:29:44.014711656 +0000 UTC m=+0.016473031 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:29:44.343 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:44 smithi016 podman[104362]: 2024-09-17 12:29:44.139941397 +0000 UTC m=+0.141702851 container create cd0e880c6dfe630ba6ae1fc320dd5f0ef29e8bdb755f3f9820a43e76411ee3c4 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-17T12:29:44.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:44 smithi016 ceph-mon[87889]: pgmap v79: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:29:44.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:44 smithi016 ceph-mon[87889]: osdmap e116: 8 total, 7 up, 8 in 2024-09-17T12:29:44.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:44 smithi016 ceph-mon[87889]: from='client.44246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:44.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:44 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/2833441987' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:29:44.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:44 smithi016 ceph-mon[89784]: pgmap v79: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:29:44.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:44 smithi016 ceph-mon[89784]: osdmap e116: 8 total, 7 up, 8 in 2024-09-17T12:29:44.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:44 smithi016 ceph-mon[89784]: from='client.44246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:44.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:44 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/2833441987' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:29:44.693 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:44 smithi016 podman[104362]: 2024-09-17 12:29:44.342229597 +0000 UTC m=+0.343990985 container init cd0e880c6dfe630ba6ae1fc320dd5f0ef29e8bdb755f3f9820a43e76411ee3c4 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, org.label-schema.license=GPLv2) 2024-09-17T12:29:44.693 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:44 smithi016 podman[104362]: 2024-09-17 12:29:44.345967539 +0000 UTC m=+0.347728915 container start cd0e880c6dfe630ba6ae1fc320dd5f0ef29e8bdb755f3f9820a43e76411ee3c4 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-7498493, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-17T12:29:44.693 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:44 smithi016 podman[104362]: 2024-09-17 12:29:44.433984086 +0000 UTC m=+0.435745472 container attach cd0e880c6dfe630ba6ae1fc320dd5f0ef29e8bdb755f3f9820a43e76411ee3c4 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-7498493, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-17T12:29:44.693 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:44 smithi016 podman[104362]: 2024-09-17 12:29:44.549251371 +0000 UTC m=+0.551012753 container died cd0e880c6dfe630ba6ae1fc320dd5f0ef29e8bdb755f3f9820a43e76411ee3c4 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-7498493, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0) 2024-09-17T12:29:45.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:44 smithi196 ceph-mon[80339]: pgmap v79: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:29:45.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:44 smithi196 ceph-mon[80339]: osdmap e116: 8 total, 7 up, 8 in 2024-09-17T12:29:45.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:44 smithi196 ceph-mon[80339]: from='client.44246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:29:45.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:44 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/2833441987' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:29:45.441 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 podman[104388]: 2024-09-17 12:29:45.138152994 +0000 UTC m=+0.582833654 container remove cd0e880c6dfe630ba6ae1fc320dd5f0ef29e8bdb755f3f9820a43e76411ee3c4 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-17T12:29:45.441 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.2.service: Deactivated successfully. 2024-09-17T12:29:45.441 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 systemd[1]: Stopped Ceph osd.2 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:29:45.442 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.2.service: Consumed 6.715s CPU time. 2024-09-17T12:29:45.442 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 systemd[1]: Starting Ceph osd.2 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:29:45.892 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:45 smithi016 ceph-mon[87889]: Health check failed: Degraded data redundancy: 51/633 objects degraded (8.057%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:45.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:45 smithi016 ceph-mon[89784]: Health check failed: Degraded data redundancy: 51/633 objects degraded (8.057%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:45.893 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 podman[104488]: 2024-09-17 12:29:45.581072581 +0000 UTC m=+0.102358095 container create d71314e2f358f23c6a416add07a678a082f500c9a05c8c16f13e82ec9787e361 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-7498493, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:29:45.893 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 podman[104488]: 2024-09-17 12:29:45.497066029 +0000 UTC m=+0.018351512 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:29:45.893 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 podman[104488]: 2024-09-17 12:29:45.816952943 +0000 UTC m=+0.338238425 container init d71314e2f358f23c6a416add07a678a082f500c9a05c8c16f13e82ec9787e361 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-7498493, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:29:45.893 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 podman[104488]: 2024-09-17 12:29:45.820400282 +0000 UTC m=+0.341685756 container start d71314e2f358f23c6a416add07a678a082f500c9a05c8c16f13e82ec9787e361 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493) 2024-09-17T12:29:45.893 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 podman[104488]: 2024-09-17 12:29:45.891702066 +0000 UTC m=+0.412987538 container attach d71314e2f358f23c6a416add07a678a082f500c9a05c8c16f13e82ec9787e361 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-7498493, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-17T12:29:46.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:45 smithi196 ceph-mon[80339]: Health check failed: Degraded data redundancy: 51/633 objects degraded (8.057%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:46.191 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate[104503]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:29:46.191 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 bash[104488]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:29:46.191 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate[104503]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:29:46.191 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:45 smithi016 bash[104488]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:29:46.443 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate[104503]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T12:29:46.444 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 bash[104488]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T12:29:46.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:29:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:29:46] "GET /metrics HTTP/1.1" 200 38557 "" "Prometheus/2.43.0" 2024-09-17T12:29:46.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:46 smithi016 ceph-mon[87889]: pgmap v81: 161 pgs: 27 active+undersized, 2 stale+active+clean, 13 active+undersized+degraded, 119 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 51/633 objects degraded (8.057%) 2024-09-17T12:29:46.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:46 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:46.726 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate[104503]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T12:29:46.726 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 bash[104488]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T12:29:46.726 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate[104503]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T12:29:46.727 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:46 smithi016 ceph-mon[89784]: pgmap v81: 161 pgs: 27 active+undersized, 2 stale+active+clean, 13 active+undersized+degraded, 119 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 51/633 objects degraded (8.057%) 2024-09-17T12:29:46.727 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:46 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:47.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:46 smithi196 ceph-mon[80339]: pgmap v81: 161 pgs: 27 active+undersized, 2 stale+active+clean, 13 active+undersized+degraded, 119 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 51/633 objects degraded (8.057%) 2024-09-17T12:29:47.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:46 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:47.048 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 bash[104488]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T12:29:47.048 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate[104503]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T12:29:47.048 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 bash[104488]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T12:29:47.048 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate[104503]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-17T12:29:47.048 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 bash[104488]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-17T12:29:47.048 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate[104503]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T12:29:47.048 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 bash[104488]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T12:29:47.049 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate[104503]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-17T12:29:47.049 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 bash[104488]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-17T12:29:47.049 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:46 smithi016 podman[104488]: 2024-09-17 12:29:46.754660026 +0000 UTC m=+1.275945498 container died d71314e2f358f23c6a416add07a678a082f500c9a05c8c16f13e82ec9787e361 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-7498493, CEPH_POINT_RELEASE=) 2024-09-17T12:29:47.691 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:47 smithi016 podman[104632]: 2024-09-17 12:29:47.397799805 +0000 UTC m=+0.636832306 container remove d71314e2f358f23c6a416add07a678a082f500c9a05c8c16f13e82ec9787e361 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=main-7498493, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-17T12:29:47.691 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:47 smithi016 podman[104690]: 2024-09-17 12:29:47.615305278 +0000 UTC m=+0.112233952 container create 99430f24b824e85c79ab33304fb9373a3eaef586c57aadccf4d8b95834dc0865 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-17T12:29:47.691 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:47 smithi016 podman[104690]: 2024-09-17 12:29:47.520688083 +0000 UTC m=+0.017616757 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:29:48.191 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:47 smithi016 podman[104690]: 2024-09-17 12:29:47.85962051 +0000 UTC m=+0.356549184 container init 99430f24b824e85c79ab33304fb9373a3eaef586c57aadccf4d8b95834dc0865 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=main-7498493) 2024-09-17T12:29:48.191 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:47 smithi016 podman[104690]: 2024-09-17 12:29:47.863105627 +0000 UTC m=+0.360034301 container start 99430f24b824e85c79ab33304fb9373a3eaef586c57aadccf4d8b95834dc0865 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-7498493, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:29:48.192 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:47 smithi016 ceph-osd[104709]: -- 172.21.15.16:0/694083737 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557414e43a40 con 0x557415c12000 2024-09-17T12:29:48.192 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:47 smithi016 bash[104690]: 99430f24b824e85c79ab33304fb9373a3eaef586c57aadccf4d8b95834dc0865 2024-09-17T12:29:48.192 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:47 smithi016 systemd[1]: Started Ceph osd.2 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:29:49.269 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:49 smithi016 ceph-mon[87889]: pgmap v82: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 57/633 objects degraded (9.005%) 2024-09-17T12:29:49.269 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:49 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:49.269 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:49 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:49.269 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:29:49.269 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:49 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:49.269 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:29:49.270 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:49 smithi016 ceph-mon[89784]: pgmap v82: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 57/633 objects degraded (9.005%) 2024-09-17T12:29:49.270 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:49 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:49.270 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:49 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:49.270 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:29:49.270 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:49 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:49.270 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:29:49.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:49 smithi196 ceph-mon[80339]: pgmap v82: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 57/633 objects degraded (9.005%) 2024-09-17T12:29:49.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:49 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:49.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:49 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:29:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:49 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:29:49.560 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:49 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2[104705]: 2024-09-17T12:29:49.266+0000 7f80a7cff640 -1 Falling back to public interface 2024-09-17T12:29:51.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:51 smithi196 ceph-mon[80339]: pgmap v83: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 57/633 objects degraded (9.005%) 2024-09-17T12:29:51.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:51 smithi016 ceph-mon[87889]: pgmap v83: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 57/633 objects degraded (9.005%) 2024-09-17T12:29:51.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:51 smithi016 ceph-mon[89784]: pgmap v83: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 57/633 objects degraded (9.005%) 2024-09-17T12:29:52.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:52 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 57/633 objects degraded (9.005%), 15 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:52.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:52 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 57/633 objects degraded (9.005%), 15 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:52.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:52 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 57/633 objects degraded (9.005%), 15 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:52.941 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:52 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2[104705]: 2024-09-17T12:29:52.595+0000 7f80a7cff640 -1 osd.2 114 log_to_monitors true 2024-09-17T12:29:53.441 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:29:53 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2[104705]: 2024-09-17T12:29:53.218+0000 7f809faa9640 -1 osd.2 114 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:29:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:53 smithi016 ceph-mon[87889]: pgmap v84: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 57/633 objects degraded (9.005%) 2024-09-17T12:29:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:53 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:53.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:53 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:53.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:53 smithi016 ceph-mon[87889]: from='osd.2 [v2:172.21.15.16:6818/2971247012,v1:172.21.15.16:6819/2971247012]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-17T12:29:53.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:53 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:53.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:53 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:53 smithi016 ceph-mon[89784]: pgmap v84: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 57/633 objects degraded (9.005%) 2024-09-17T12:29:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:53 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:53 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:53 smithi016 ceph-mon[89784]: from='osd.2 [v2:172.21.15.16:6818/2971247012,v1:172.21.15.16:6819/2971247012]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-17T12:29:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:53 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:53 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:53.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:53 smithi196 ceph-mon[80339]: pgmap v84: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 57/633 objects degraded (9.005%) 2024-09-17T12:29:53.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:53 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:53.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:53 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:53.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:53 smithi196 ceph-mon[80339]: from='osd.2 [v2:172.21.15.16:6818/2971247012,v1:172.21.15.16:6819/2971247012]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-17T12:29:53.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:53 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:53.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:53 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:54.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:54 smithi016 ceph-mon[87889]: from='osd.2 [v2:172.21.15.16:6818/2971247012,v1:172.21.15.16:6819/2971247012]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T12:29:54.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:54 smithi016 ceph-mon[87889]: osdmap e117: 8 total, 7 up, 8 in 2024-09-17T12:29:54.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:54 smithi016 ceph-mon[87889]: from='osd.2 [v2:172.21.15.16:6818/2971247012,v1:172.21.15.16:6819/2971247012]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:29:54.388 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:54 smithi016 ceph-mon[89784]: from='osd.2 [v2:172.21.15.16:6818/2971247012,v1:172.21.15.16:6819/2971247012]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T12:29:54.388 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:54 smithi016 ceph-mon[89784]: osdmap e117: 8 total, 7 up, 8 in 2024-09-17T12:29:54.388 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:54 smithi016 ceph-mon[89784]: from='osd.2 [v2:172.21.15.16:6818/2971247012,v1:172.21.15.16:6819/2971247012]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:29:54.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:54 smithi196 ceph-mon[80339]: from='osd.2 [v2:172.21.15.16:6818/2971247012,v1:172.21.15.16:6819/2971247012]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T12:29:54.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:54 smithi196 ceph-mon[80339]: osdmap e117: 8 total, 7 up, 8 in 2024-09-17T12:29:54.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:54 smithi196 ceph-mon[80339]: from='osd.2 [v2:172.21.15.16:6818/2971247012,v1:172.21.15.16:6819/2971247012]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:29:55.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:55 smithi016 ceph-mon[87889]: pgmap v86: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 57/633 objects degraded (9.005%) 2024-09-17T12:29:55.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:55 smithi016 ceph-mon[87889]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:29:55.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:55 smithi016 ceph-mon[87889]: osd.2 [v2:172.21.15.16:6818/2971247012,v1:172.21.15.16:6819/2971247012] boot 2024-09-17T12:29:55.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:55 smithi016 ceph-mon[87889]: osdmap e118: 8 total, 8 up, 8 in 2024-09-17T12:29:55.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:55 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T12:29:55.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:55 smithi016 ceph-mon[87889]: osdmap e119: 8 total, 8 up, 8 in 2024-09-17T12:29:55.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:55 smithi016 ceph-mon[89784]: pgmap v86: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 57/633 objects degraded (9.005%) 2024-09-17T12:29:55.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:55 smithi016 ceph-mon[89784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:29:55.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:55 smithi016 ceph-mon[89784]: osd.2 [v2:172.21.15.16:6818/2971247012,v1:172.21.15.16:6819/2971247012] boot 2024-09-17T12:29:55.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:55 smithi016 ceph-mon[89784]: osdmap e118: 8 total, 8 up, 8 in 2024-09-17T12:29:55.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:55 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T12:29:55.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:55 smithi016 ceph-mon[89784]: osdmap e119: 8 total, 8 up, 8 in 2024-09-17T12:29:55.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:55 smithi196 ceph-mon[80339]: pgmap v86: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 57/633 objects degraded (9.005%) 2024-09-17T12:29:55.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:55 smithi196 ceph-mon[80339]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:29:55.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:55 smithi196 ceph-mon[80339]: osd.2 [v2:172.21.15.16:6818/2971247012,v1:172.21.15.16:6819/2971247012] boot 2024-09-17T12:29:55.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:55 smithi196 ceph-mon[80339]: osdmap e118: 8 total, 8 up, 8 in 2024-09-17T12:29:55.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:55 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T12:29:55.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:55 smithi196 ceph-mon[80339]: osdmap e119: 8 total, 8 up, 8 in 2024-09-17T12:29:56.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:29:55 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:29:55.909+0000 7f9d37c73640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-09-17T12:29:56.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:29:56] "GET /metrics HTTP/1.1" 200 38639 "" "Prometheus/2.43.0" 2024-09-17T12:29:57.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: pgmap v89: 161 pgs: 24 peering, 15 active+undersized, 3 active+undersized+degraded, 119 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 7/633 objects degraded (1.106%) 2024-09-17T12:29:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:29:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:29:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:29:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-17T12:29:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T12:29:57.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-09-17T12:29:57.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:57.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 7/633 objects degraded (1.106%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:57.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: pgmap v89: 161 pgs: 24 peering, 15 active+undersized, 3 active+undersized+degraded, 119 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 7/633 objects degraded (1.106%) 2024-09-17T12:29:57.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:57.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:57.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:29:57.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:29:57.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:57.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:29:57.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:57.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:57.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:57.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-17T12:29:57.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T12:29:57.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-09-17T12:29:57.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:57.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:56 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 7/633 objects degraded (1.106%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:57.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: pgmap v89: 161 pgs: 24 peering, 15 active+undersized, 3 active+undersized+degraded, 119 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 7/633 objects degraded (1.106%) 2024-09-17T12:29:57.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:57.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:57.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:29:57.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:29:57.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:29:57.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:29:57.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:57.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:57.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:29:57.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-17T12:29:57.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T12:29:57.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-09-17T12:29:57.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:29:57.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:56 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 7/633 objects degraded (1.106%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T12:29:59.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:58 smithi016 ceph-mon[87889]: pgmap v90: 161 pgs: 24 peering, 11 active+undersized, 2 active+undersized+degraded, 124 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/633 objects degraded (0.316%) 2024-09-17T12:29:59.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:58 smithi016 ceph-mon[89784]: pgmap v90: 161 pgs: 24 peering, 11 active+undersized, 2 active+undersized+degraded, 124 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/633 objects degraded (0.316%) 2024-09-17T12:29:59.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:58 smithi196 ceph-mon[80339]: pgmap v90: 161 pgs: 24 peering, 11 active+undersized, 2 active+undersized+degraded, 124 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/633 objects degraded (0.316%) 2024-09-17T12:30:00.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:59 smithi016 ceph-mon[87889]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/633 objects degraded (0.316%), 2 pgs degraded) 2024-09-17T12:30:00.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:29:59 smithi016 ceph-mon[87889]: Cluster is now healthy 2024-09-17T12:30:00.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:59 smithi016 ceph-mon[89784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/633 objects degraded (0.316%), 2 pgs degraded) 2024-09-17T12:30:00.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:29:59 smithi016 ceph-mon[89784]: Cluster is now healthy 2024-09-17T12:30:00.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:59 smithi196 ceph-mon[80339]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/633 objects degraded (0.316%), 2 pgs degraded) 2024-09-17T12:30:00.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:29:59 smithi196 ceph-mon[80339]: Cluster is now healthy 2024-09-17T12:30:01.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:00 smithi016 ceph-mon[87889]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-17T12:30:01.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:00 smithi016 ceph-mon[87889]: overall HEALTH_OK 2024-09-17T12:30:01.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:00 smithi016 ceph-mon[89784]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-17T12:30:01.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:00 smithi016 ceph-mon[89784]: overall HEALTH_OK 2024-09-17T12:30:01.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:00 smithi196 ceph-mon[80339]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-17T12:30:01.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:00 smithi196 ceph-mon[80339]: overall HEALTH_OK 2024-09-17T12:30:03.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:02 smithi016 ceph-mon[87889]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:30:03.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:02 smithi016 ceph-mon[89784]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:30:03.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:02 smithi196 ceph-mon[80339]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:30:05.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:04 smithi016 ceph-mon[87889]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:30:05.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:04 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:05.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:04 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:30:05.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:04 smithi016 ceph-mon[89784]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:30:05.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:04 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:05.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:04 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:30:05.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:04 smithi196 ceph-mon[80339]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:30:05.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:04 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:05.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:04 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:30:06.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:30:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:30:06] "GET /metrics HTTP/1.1" 200 38630 "" "Prometheus/2.43.0" 2024-09-17T12:30:07.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:06 smithi016 ceph-mon[87889]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2024-09-17T12:30:07.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:06 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:07.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:06 smithi016 ceph-mon[89784]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2024-09-17T12:30:07.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:06 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:07.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:06 smithi196 ceph-mon[80339]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2024-09-17T12:30:07.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:06 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:08 smithi016 ceph-mon[87889]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:30:09.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:08 smithi016 ceph-mon[89784]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:30:09.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:08 smithi196 ceph-mon[80339]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:30:10.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:09 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:10.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:09 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:10.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:09 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:11.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:10 smithi196 ceph-mon[80339]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:30:11.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:10 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-17T12:30:11.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:10 smithi016 ceph-mon[87889]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:30:11.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:10 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-17T12:30:11.298 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:10 smithi016 ceph-mon[89784]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:30:11.298 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:10 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-17T12:30:12.245 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:11 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T12:30:12.246 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:11 smithi016 ceph-mon[87889]: Upgrade: osd.3 is safe to restart 2024-09-17T12:30:12.246 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:11 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T12:30:12.246 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:11 smithi016 ceph-mon[89784]: Upgrade: osd.3 is safe to restart 2024-09-17T12:30:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:11 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T12:30:12.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:11 smithi196 ceph-mon[80339]: Upgrade: osd.3 is safe to restart 2024-09-17T12:30:13.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:12 smithi016 ceph-mon[87889]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:30:13.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:12 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:13.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:12 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-17T12:30:13.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:12 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:30:13.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:12 smithi016 ceph-mon[89784]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:30:13.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:12 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:13.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:12 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-17T12:30:13.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:12 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:30:13.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:12 smithi196 ceph-mon[80339]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:30:13.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:12 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:13.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:12 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-17T12:30:13.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:12 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:30:14.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:13 smithi196 ceph-mon[80339]: Upgrade: Updating osd.3 2024-09-17T12:30:14.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:13 smithi196 ceph-mon[80339]: Deploying daemon osd.3 on smithi016 2024-09-17T12:30:14.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:13 smithi016 ceph-mon[87889]: Upgrade: Updating osd.3 2024-09-17T12:30:14.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:13 smithi016 ceph-mon[87889]: Deploying daemon osd.3 on smithi016 2024-09-17T12:30:14.287 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:13 smithi016 ceph-mon[89784]: Upgrade: Updating osd.3 2024-09-17T12:30:14.287 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:13 smithi016 ceph-mon[89784]: Deploying daemon osd.3 on smithi016 2024-09-17T12:30:14.409 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:30:15.007 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:30:15.008 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (8m) 22s ago 9m 25.0M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:30:15.008 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (8m) 2m ago 8m 89.2M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:30:15.008 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (3m) 22s ago 7m 60.8M - 3.8 980a0bfcd684 843db1407344 2024-09-17T12:30:15.008 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (3m) 2m ago 13m 453M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:30:15.008 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (3m) 22s ago 14m 517M - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:30:15.008 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (2m) 22s ago 14m 47.0M 2048M 19.3.0-4937-g74984932 6c1fcab85602 4a83b787c605 2024-09-17T12:30:15.008 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (2m) 2m ago 13m 24.7M 2048M 19.3.0-4937-g74984932 6c1fcab85602 dc48dfc9e40c 2024-09-17T12:30:15.008 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (2m) 22s ago 13m 41.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 aa5074acd22a 2024-09-17T12:30:15.008 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (8m) 22s ago 8m 15.4M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:30:15.009 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (8m) 2m ago 8m 15.4M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:30:15.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (94s) 22s ago 12m 43.7M 2737M 19.3.0-4937-g74984932 6c1fcab85602 d12e6f26f048 2024-09-17T12:30:15.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (60s) 22s ago 12m 52.5M 2737M 19.3.0-4937-g74984932 6c1fcab85602 cd2f6e2964dd 2024-09-17T12:30:15.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (27s) 22s ago 11m 12.0M 2737M 19.3.0-4937-g74984932 6c1fcab85602 99430f24b824 2024-09-17T12:30:15.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (11m) 22s ago 11m 63.5M 2737M 18.2.4-855-g19e2a2be 980a0bfcd684 4a4f8eeaff13 2024-09-17T12:30:15.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (11m) 2m ago 11m 55.7M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 afa20056b398 2024-09-17T12:30:15.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (10m) 2m ago 10m 51.2M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 f0dcabb8902f 2024-09-17T12:30:15.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (10m) 2m ago 10m 60.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:30:15.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (9m) 2m ago 9m 61.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:30:15.009 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (3m) 2m ago 9m 42.7M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:30:15.009 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (7m) 22s ago 7m 80.5M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:30:15.009 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (7m) 2m ago 7m 80.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:30:15.010 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (7m) 22s ago 7m 82.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:30:15.010 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (7m) 2m ago 7m 79.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:30:15.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:14 smithi016 ceph-mon[87889]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:30:15.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:14 smithi016 ceph-mon[89784]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:30:15.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:14 smithi196 ceph-mon[80339]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:30:15.398 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:30:15.398 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:30:15.398 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:30:15.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:30:15.398 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:30:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:30:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:30:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:30:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 5, 2024-09-17T12:30:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:30:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:30:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:30:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:30:15.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:30:15.400 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:30:15.400 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 9, 2024-09-17T12:30:15.400 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 8 2024-09-17T12:30:15.400 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:30:15.400 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:30:15.579 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:15 smithi016 systemd[1]: Stopping Ceph osd.3 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:30:15.724 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:30:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:30:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:30:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:30:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:30:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-09-17T12:30:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:30:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:30:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "8/23 daemons upgraded", 2024-09-17T12:30:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T12:30:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:30:15.725 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:30:15.941 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:15 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3[56177]: 2024-09-17T12:30:15.577+0000 7fd56405e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:30:15.941 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:15 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3[56177]: 2024-09-17T12:30:15.577+0000 7fd56405e640 -1 osd.3 119 *** Got signal Terminated *** 2024-09-17T12:30:15.941 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:15 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3[56177]: 2024-09-17T12:30:15.577+0000 7fd56405e640 -1 osd.3 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:30:16.086 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 osds down 2024-09-17T12:30:16.086 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-17T12:30:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: osd.3 (root=default,host=smithi016) is down 2024-09-17T12:30:16.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:16 smithi196 ceph-mon[80339]: from='client.34329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:16.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:16 smithi196 ceph-mon[80339]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:16.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:16 smithi196 ceph-mon[80339]: from='client.44276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:16.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:16 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/3024746093' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:16.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:16 smithi196 ceph-mon[80339]: osd.3 marked itself down and dead 2024-09-17T12:30:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:16 smithi016 ceph-mon[87889]: from='client.34329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:16 smithi016 ceph-mon[87889]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:16 smithi016 ceph-mon[87889]: from='client.44276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:16 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/3024746093' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:16 smithi016 ceph-mon[87889]: osd.3 marked itself down and dead 2024-09-17T12:30:16.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:30:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:30:16] "GET /metrics HTTP/1.1" 200 38630 "" "Prometheus/2.43.0" 2024-09-17T12:30:16.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:16 smithi016 ceph-mon[89784]: from='client.34329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:16.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:16 smithi016 ceph-mon[89784]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:16.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:16 smithi016 ceph-mon[89784]: from='client.44276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:16.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:16 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/3024746093' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:16.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:16 smithi016 ceph-mon[89784]: osd.3 marked itself down and dead 2024-09-17T12:30:16.691 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:16 smithi016 podman[109064]: 2024-09-17 12:30:16.411879024 +0000 UTC m=+0.958758237 container died 4a4f8eeaff1348b4f98556d907975e512473bc668d9cd4f49763bbbb97ecb8ac (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-17T12:30:17.005 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:16 smithi016 podman[109064]: 2024-09-17 12:30:16.801392947 +0000 UTC m=+1.348272148 container cleanup 4a4f8eeaff1348b4f98556d907975e512473bc668d9cd4f49763bbbb97ecb8ac (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:30:17.005 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:16 smithi016 bash[109064]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3 2024-09-17T12:30:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:17 smithi196 ceph-mon[80339]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:30:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:17 smithi196 ceph-mon[80339]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:17 smithi196 ceph-mon[80339]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:30:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:17 smithi196 ceph-mon[80339]: osdmap e120: 8 total, 7 up, 8 in 2024-09-17T12:30:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:17 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/1818512264' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:30:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:17 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:17.382 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:17 smithi016 podman[109124]: 2024-09-17 12:30:17.044509794 +0000 UTC m=+0.627563242 container remove 4a4f8eeaff1348b4f98556d907975e512473bc668d9cd4f49763bbbb97ecb8ac (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-09-17T12:30:17.382 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:17 smithi016 ceph-mon[89784]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:30:17.382 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:17 smithi016 ceph-mon[89784]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:17.382 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:17 smithi016 ceph-mon[89784]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:30:17.382 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:17 smithi016 ceph-mon[89784]: osdmap e120: 8 total, 7 up, 8 in 2024-09-17T12:30:17.383 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:17 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/1818512264' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:30:17.383 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:17 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:17.383 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:17 smithi016 ceph-mon[87889]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:30:17.383 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:17 smithi016 ceph-mon[87889]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:17.383 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:17 smithi016 ceph-mon[87889]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:30:17.383 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:17 smithi016 ceph-mon[87889]: osdmap e120: 8 total, 7 up, 8 in 2024-09-17T12:30:17.383 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:17 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/1818512264' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:30:17.383 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:17 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:17.691 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:17 smithi016 podman[109214]: 2024-09-17 12:30:17.283715188 +0000 UTC m=+0.016566117 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:30:17.691 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:17 smithi016 podman[109214]: 2024-09-17 12:30:17.403721072 +0000 UTC m=+0.136572057 container create ed92636ff760b48b4edeee7274f85138d89064f1579b91af6df2a499be190692 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-7498493, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:30:17.691 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:17 smithi016 podman[109214]: 2024-09-17 12:30:17.623004153 +0000 UTC m=+0.355855093 container init ed92636ff760b48b4edeee7274f85138d89064f1579b91af6df2a499be190692 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-7498493, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-17T12:30:17.691 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:17 smithi016 podman[109214]: 2024-09-17 12:30:17.626494276 +0000 UTC m=+0.359345199 container start ed92636ff760b48b4edeee7274f85138d89064f1579b91af6df2a499be190692 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-7498493, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD) 2024-09-17T12:30:18.015 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:17 smithi016 podman[109214]: 2024-09-17 12:30:17.697745123 +0000 UTC m=+0.430596044 container attach ed92636ff760b48b4edeee7274f85138d89064f1579b91af6df2a499be190692 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-7498493, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-17T12:30:18.015 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:17 smithi016 podman[109214]: 2024-09-17 12:30:17.832247431 +0000 UTC m=+0.565098352 container died ed92636ff760b48b4edeee7274f85138d89064f1579b91af6df2a499be190692 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, ceph=True, RELEASE=main-7498493, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-17T12:30:18.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:18 smithi196 ceph-mon[80339]: osdmap e121: 8 total, 7 up, 8 in 2024-09-17T12:30:18.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:18 smithi016 ceph-mon[87889]: osdmap e121: 8 total, 7 up, 8 in 2024-09-17T12:30:18.369 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:18 smithi016 ceph-mon[89784]: osdmap e121: 8 total, 7 up, 8 in 2024-09-17T12:30:18.691 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:18 smithi016 podman[109237]: 2024-09-17 12:30:18.368152256 +0000 UTC m=+0.525978242 container remove ed92636ff760b48b4edeee7274f85138d89064f1579b91af6df2a499be190692 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-7498493, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:30:18.691 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:18 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.3.service: Deactivated successfully. 2024-09-17T12:30:18.691 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:18 smithi016 systemd[1]: Stopped Ceph osd.3 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:30:18.691 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:18 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.3.service: Consumed 7.976s CPU time. 2024-09-17T12:30:18.691 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:18 smithi016 systemd[1]: Starting Ceph osd.3 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:30:19.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-mon[87889]: pgmap v102: 161 pgs: 7 active+undersized, 21 stale+active+clean, 7 active+undersized+degraded, 126 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 24/633 objects degraded (3.791%) 2024-09-17T12:30:19.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-mon[87889]: Health check failed: Degraded data redundancy: 24/633 objects degraded (3.791%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T12:30:19.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:19.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:30:19.106 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-mon[89784]: pgmap v102: 161 pgs: 7 active+undersized, 21 stale+active+clean, 7 active+undersized+degraded, 126 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 24/633 objects degraded (3.791%) 2024-09-17T12:30:19.106 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-mon[89784]: Health check failed: Degraded data redundancy: 24/633 objects degraded (3.791%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T12:30:19.106 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:19.106 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:30:19.107 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:18 smithi016 podman[109335]: 2024-09-17 12:30:18.819031274 +0000 UTC m=+0.102767824 container create cec4d670c9cbd6fbce60e2ae37f00ace5808d22fbc383f6951eb925bb7c727fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-7498493, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-17T12:30:19.107 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:18 smithi016 podman[109335]: 2024-09-17 12:30:18.732657807 +0000 UTC m=+0.016394355 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:30:19.107 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 podman[109335]: 2024-09-17 12:30:19.03040756 +0000 UTC m=+0.314144114 container init cec4d670c9cbd6fbce60e2ae37f00ace5808d22fbc383f6951eb925bb7c727fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , RELEASE=main-7498493, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-09-17T12:30:19.107 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 podman[109335]: 2024-09-17 12:30:19.034550269 +0000 UTC m=+0.318286825 container start cec4d670c9cbd6fbce60e2ae37f00ace5808d22fbc383f6951eb925bb7c727fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-7498493, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-17T12:30:19.107 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 podman[109335]: 2024-09-17 12:30:19.104892563 +0000 UTC m=+0.388629115 container attach cec4d670c9cbd6fbce60e2ae37f00ace5808d22fbc383f6951eb925bb7c727fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-7498493, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-17T12:30:19.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:19 smithi196 ceph-mon[80339]: pgmap v102: 161 pgs: 7 active+undersized, 21 stale+active+clean, 7 active+undersized+degraded, 126 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 24/633 objects degraded (3.791%) 2024-09-17T12:30:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:19 smithi196 ceph-mon[80339]: Health check failed: Degraded data redundancy: 24/633 objects degraded (3.791%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T12:30:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:19 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:19 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:30:19.441 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate[109350]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:30:19.441 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 bash[109335]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:30:19.441 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate[109350]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:30:19.441 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 bash[109335]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:30:19.939 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate[109350]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T12:30:19.939 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 bash[109335]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T12:30:19.939 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate[109350]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T12:30:19.939 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 bash[109335]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T12:30:19.939 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate[109350]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T12:30:19.939 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 bash[109335]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T12:30:20.191 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate[109350]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T12:30:20.191 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 bash[109335]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T12:30:20.191 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate[109350]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-17T12:30:20.191 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 bash[109335]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-17T12:30:20.192 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate[109350]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T12:30:20.192 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 bash[109335]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T12:30:20.192 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate[109350]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-17T12:30:20.192 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 bash[109335]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-17T12:30:20.192 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:19 smithi016 podman[109335]: 2024-09-17 12:30:19.969242793 +0000 UTC m=+1.252979341 container died cec4d670c9cbd6fbce60e2ae37f00ace5808d22fbc383f6951eb925bb7c727fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate, io.buildah.version=1.37.2, ceph=True, RELEASE=main-7498493, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:30:20.868 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:20 smithi016 podman[109479]: 2024-09-17 12:30:20.55882618 +0000 UTC m=+0.583572038 container remove cec4d670c9cbd6fbce60e2ae37f00ace5808d22fbc383f6951eb925bb7c727fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-7498493, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-17T12:30:20.868 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:20 smithi016 podman[109539]: 2024-09-17 12:30:20.768222642 +0000 UTC m=+0.107707439 container create 1d4309778a0d2596b6d7276f16e373eb1dfa17a1e93b97e23ce883c74d609e24 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-7498493, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-17T12:30:20.868 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:20 smithi016 podman[109539]: 2024-09-17 12:30:20.677225698 +0000 UTC m=+0.016710491 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:30:21.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:21 smithi016 ceph-mon[87889]: pgmap v103: 161 pgs: 33 active+undersized, 6 stale+active+clean, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 63/633 objects degraded (9.953%) 2024-09-17T12:30:21.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:21 smithi016 ceph-mon[89784]: pgmap v103: 161 pgs: 33 active+undersized, 6 stale+active+clean, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 63/633 objects degraded (9.953%) 2024-09-17T12:30:21.191 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:20 smithi016 podman[109539]: 2024-09-17 12:30:20.98708505 +0000 UTC m=+0.326569855 container init 1d4309778a0d2596b6d7276f16e373eb1dfa17a1e93b97e23ce883c74d609e24 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-7498493, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-17T12:30:21.192 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:20 smithi016 podman[109539]: 2024-09-17 12:30:20.990409194 +0000 UTC m=+0.329893988 container start 1d4309778a0d2596b6d7276f16e373eb1dfa17a1e93b97e23ce883c74d609e24 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=main-7498493, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-17T12:30:21.192 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:21 smithi016 ceph-osd[109558]: -- 172.21.15.16:0/3187876671 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557cb84bda40 con 0x557cb9290000 2024-09-17T12:30:21.192 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:21 smithi016 bash[109539]: 1d4309778a0d2596b6d7276f16e373eb1dfa17a1e93b97e23ce883c74d609e24 2024-09-17T12:30:21.192 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:21 smithi016 systemd[1]: Started Ceph osd.3 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:30:21.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:21 smithi196 ceph-mon[80339]: pgmap v103: 161 pgs: 33 active+undersized, 6 stale+active+clean, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 63/633 objects degraded (9.953%) 2024-09-17T12:30:22.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:22 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:22.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:22 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:22.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:22 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:30:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:22 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:22 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:22.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:22 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:30:22.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:22 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:22.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:22 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:22.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:22 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:30:22.941 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:22 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3[109554]: 2024-09-17T12:30:22.874+0000 7fa325aca640 -1 Falling back to public interface 2024-09-17T12:30:23.398 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:23 smithi016 ceph-mon[87889]: pgmap v104: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 81/633 objects degraded (12.796%) 2024-09-17T12:30:23.398 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:23 smithi016 ceph-mon[89784]: pgmap v104: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 81/633 objects degraded (12.796%) 2024-09-17T12:30:23.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:23 smithi196 ceph-mon[80339]: pgmap v104: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 81/633 objects degraded (12.796%) 2024-09-17T12:30:25.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:25 smithi016 ceph-mon[87889]: pgmap v105: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 81/633 objects degraded (12.796%) 2024-09-17T12:30:25.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:25 smithi016 ceph-mon[89784]: pgmap v105: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 81/633 objects degraded (12.796%) 2024-09-17T12:30:25.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:25 smithi196 ceph-mon[80339]: pgmap v105: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 81/633 objects degraded (12.796%) 2024-09-17T12:30:26.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:26.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:26.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-mon[87889]: pgmap v106: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 81/633 objects degraded (12.796%) 2024-09-17T12:30:26.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:26.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:26.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:26.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:30:26] "GET /metrics HTTP/1.1" 200 38636 "" "Prometheus/2.43.0" 2024-09-17T12:30:26.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:26.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:26.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-mon[89784]: pgmap v106: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 81/633 objects degraded (12.796%) 2024-09-17T12:30:26.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:26.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:26.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:26.676 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3[109554]: 2024-09-17T12:30:26.427+0000 7fa325aca640 -1 osd.3 119 log_to_monitors true 2024-09-17T12:30:26.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:26 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:26.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:26 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:26.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:26 smithi196 ceph-mon[80339]: pgmap v106: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 81/633 objects degraded (12.796%) 2024-09-17T12:30:26.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:26 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:26.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:26 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:26.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:26 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:27.600 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:30:27 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3[109554]: 2024-09-17T12:30:27.376+0000 7fa31d874640 -1 osd.3 119 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:30:27.600 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:27 smithi016 ceph-mon[87889]: from='osd.3 [v2:172.21.15.16:6826/1808544095,v1:172.21.15.16:6827/1808544095]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-17T12:30:27.600 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:27 smithi016 ceph-mon[87889]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-17T12:30:27.600 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:27 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 81/633 objects degraded (12.796%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T12:30:27.600 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:27 smithi016 ceph-mon[89784]: from='osd.3 [v2:172.21.15.16:6826/1808544095,v1:172.21.15.16:6827/1808544095]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-17T12:30:27.601 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:27 smithi016 ceph-mon[89784]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-17T12:30:27.601 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:27 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 81/633 objects degraded (12.796%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T12:30:27.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:27 smithi196 ceph-mon[80339]: from='osd.3 [v2:172.21.15.16:6826/1808544095,v1:172.21.15.16:6827/1808544095]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-17T12:30:27.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:27 smithi196 ceph-mon[80339]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-17T12:30:27.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:27 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 81/633 objects degraded (12.796%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T12:30:28.570 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:28 smithi016 ceph-mon[87889]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T12:30:28.570 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:28 smithi016 ceph-mon[87889]: osdmap e122: 8 total, 7 up, 8 in 2024-09-17T12:30:28.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:28 smithi016 ceph-mon[87889]: from='osd.3 [v2:172.21.15.16:6826/1808544095,v1:172.21.15.16:6827/1808544095]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:30:28.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:28 smithi016 ceph-mon[87889]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:30:28.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:28 smithi016 ceph-mon[87889]: pgmap v108: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 81/633 objects degraded (12.796%) 2024-09-17T12:30:28.571 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:28 smithi016 ceph-mon[89784]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T12:30:28.571 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:28 smithi016 ceph-mon[89784]: osdmap e122: 8 total, 7 up, 8 in 2024-09-17T12:30:28.571 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:28 smithi016 ceph-mon[89784]: from='osd.3 [v2:172.21.15.16:6826/1808544095,v1:172.21.15.16:6827/1808544095]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:30:28.571 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:28 smithi016 ceph-mon[89784]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:30:28.571 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:28 smithi016 ceph-mon[89784]: pgmap v108: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 81/633 objects degraded (12.796%) 2024-09-17T12:30:28.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:28 smithi196 ceph-mon[80339]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T12:30:28.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:28 smithi196 ceph-mon[80339]: osdmap e122: 8 total, 7 up, 8 in 2024-09-17T12:30:28.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:28 smithi196 ceph-mon[80339]: from='osd.3 [v2:172.21.15.16:6826/1808544095,v1:172.21.15.16:6827/1808544095]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:30:28.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:28 smithi196 ceph-mon[80339]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-17T12:30:28.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:28 smithi196 ceph-mon[80339]: pgmap v108: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 81/633 objects degraded (12.796%) 2024-09-17T12:30:29.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:30:28 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:30:28.955+0000 7f9d37c73640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-17T12:30:29.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:30:29.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: osd.3 [v2:172.21.15.16:6826/1808544095,v1:172.21.15.16:6827/1808544095] boot 2024-09-17T12:30:29.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: osdmap e123: 8 total, 8 up, 8 in 2024-09-17T12:30:29.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T12:30:29.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:29.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:29.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:30:29.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:30:29.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:29.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:30:29.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:29.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:29.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:29.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-17T12:30:29.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T12:30:29.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[87889]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-17T12:30:29.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:30:29.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: osd.3 [v2:172.21.15.16:6826/1808544095,v1:172.21.15.16:6827/1808544095] boot 2024-09-17T12:30:29.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: osdmap e123: 8 total, 8 up, 8 in 2024-09-17T12:30:29.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T12:30:29.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:29.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:29.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:30:29.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:30:29.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:29.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:30:29.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:29.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:29.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:29.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-17T12:30:29.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T12:30:29.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:29 smithi016 ceph-mon[89784]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-17T12:30:29.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:30:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: osd.3 [v2:172.21.15.16:6826/1808544095,v1:172.21.15.16:6827/1808544095] boot 2024-09-17T12:30:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: osdmap e123: 8 total, 8 up, 8 in 2024-09-17T12:30:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T12:30:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:30:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:30:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:30:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:29.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:29.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-17T12:30:29.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T12:30:29.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:29 smithi196 ceph-mon[80339]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-17T12:30:30.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:30 smithi016 ceph-mon[87889]: osdmap e124: 8 total, 8 up, 8 in 2024-09-17T12:30:30.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:30 smithi016 ceph-mon[87889]: pgmap v111: 161 pgs: 29 active+undersized, 19 active+undersized+degraded, 113 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/633 objects degraded (9.479%) 2024-09-17T12:30:30.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:30 smithi016 ceph-mon[89784]: osdmap e124: 8 total, 8 up, 8 in 2024-09-17T12:30:30.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:30 smithi016 ceph-mon[89784]: pgmap v111: 161 pgs: 29 active+undersized, 19 active+undersized+degraded, 113 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/633 objects degraded (9.479%) 2024-09-17T12:30:30.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:30 smithi196 ceph-mon[80339]: osdmap e124: 8 total, 8 up, 8 in 2024-09-17T12:30:30.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:30 smithi196 ceph-mon[80339]: pgmap v111: 161 pgs: 29 active+undersized, 19 active+undersized+degraded, 113 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/633 objects degraded (9.479%) 2024-09-17T12:30:32.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:32 smithi016 ceph-mon[87889]: pgmap v112: 161 pgs: 26 active+undersized, 15 active+undersized+degraded, 120 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 50/633 objects degraded (7.899%) 2024-09-17T12:30:32.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:32 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 50/633 objects degraded (7.899%), 15 pgs degraded (PG_DEGRADED) 2024-09-17T12:30:32.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:32 smithi016 ceph-mon[89784]: pgmap v112: 161 pgs: 26 active+undersized, 15 active+undersized+degraded, 120 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 50/633 objects degraded (7.899%) 2024-09-17T12:30:32.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:32 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 50/633 objects degraded (7.899%), 15 pgs degraded (PG_DEGRADED) 2024-09-17T12:30:33.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:32 smithi196 ceph-mon[80339]: pgmap v112: 161 pgs: 26 active+undersized, 15 active+undersized+degraded, 120 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 50/633 objects degraded (7.899%) 2024-09-17T12:30:33.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:32 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 50/633 objects degraded (7.899%), 15 pgs degraded (PG_DEGRADED) 2024-09-17T12:30:33.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:33 smithi016 ceph-mon[87889]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/633 objects degraded (7.899%), 15 pgs degraded) 2024-09-17T12:30:33.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:33 smithi016 ceph-mon[87889]: Cluster is now healthy 2024-09-17T12:30:33.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:33 smithi016 ceph-mon[89784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/633 objects degraded (7.899%), 15 pgs degraded) 2024-09-17T12:30:33.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:33 smithi016 ceph-mon[89784]: Cluster is now healthy 2024-09-17T12:30:34.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:33 smithi196 ceph-mon[80339]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/633 objects degraded (7.899%), 15 pgs degraded) 2024-09-17T12:30:34.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:33 smithi196 ceph-mon[80339]: Cluster is now healthy 2024-09-17T12:30:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:34 smithi016 ceph-mon[87889]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:34 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:30:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:34 smithi016 ceph-mon[89784]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:34 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:34 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:30:35.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:34 smithi196 ceph-mon[80339]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:34 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:34 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:30:36.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:30:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:30:36] "GET /metrics HTTP/1.1" 200 38625 "" "Prometheus/2.43.0" 2024-09-17T12:30:37.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:36 smithi016 ceph-mon[87889]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:37.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:36 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:37.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:36 smithi016 ceph-mon[89784]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:36 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:37.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:36 smithi196 ceph-mon[80339]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:37.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:36 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:39.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:38 smithi016 ceph-mon[87889]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:39.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:38 smithi016 ceph-mon[89784]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:39.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:38 smithi196 ceph-mon[80339]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:40.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:40 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:40.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:39 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:40.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:39 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:41.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:41 smithi196 ceph-mon[80339]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:41.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:40 smithi016 ceph-mon[87889]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:41.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:40 smithi016 ceph-mon[89784]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:43.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:43 smithi196 ceph-mon[80339]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:43.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:43 smithi016 ceph-mon[89784]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:43.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:43 smithi016 ceph-mon[87889]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-17T12:30:44.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-17T12:30:44.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-17T12:30:44.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-17T12:30:45.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:45 smithi196 ceph-mon[80339]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T12:30:45.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:45 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T12:30:45.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:45 smithi196 ceph-mon[80339]: Upgrade: osd.4 is safe to restart 2024-09-17T12:30:45.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:45 smithi016 ceph-mon[87889]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T12:30:45.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:45 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T12:30:45.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:45 smithi016 ceph-mon[87889]: Upgrade: osd.4 is safe to restart 2024-09-17T12:30:45.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:45 smithi016 ceph-mon[89784]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T12:30:45.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:45 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T12:30:45.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:45 smithi016 ceph-mon[89784]: Upgrade: osd.4 is safe to restart 2024-09-17T12:30:46.413 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:30:46] "GET /metrics HTTP/1.1" 200 38625 "" "Prometheus/2.43.0" 2024-09-17T12:30:46.414 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:30:46.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[87889]: Upgrade: Updating osd.4 2024-09-17T12:30:46.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:46.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-17T12:30:46.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:30:46.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[87889]: Deploying daemon osd.4 on smithi196 2024-09-17T12:30:46.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[87889]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:30:46.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:46.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[89784]: Upgrade: Updating osd.4 2024-09-17T12:30:46.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:46.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-17T12:30:46.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:30:46.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[89784]: Deploying daemon osd.4 on smithi196 2024-09-17T12:30:46.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[89784]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:30:46.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:46 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:46.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:46 smithi196 ceph-mon[80339]: Upgrade: Updating osd.4 2024-09-17T12:30:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:46 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:46 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-17T12:30:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:46 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:30:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:46 smithi196 ceph-mon[80339]: Deploying daemon osd.4 on smithi196 2024-09-17T12:30:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:46 smithi196 ceph-mon[80339]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:30:46.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:46 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:47.018 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:30:47.018 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (8m) 21s ago 9m 25.0M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:30:47.018 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (9m) 2m ago 9m 89.2M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:30:47.018 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (3m) 21s ago 8m 60.9M - 3.8 980a0bfcd684 843db1407344 2024-09-17T12:30:47.018 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (3m) 2m ago 13m 453M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:30:47.018 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (4m) 21s ago 15m 519M - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:30:47.018 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (3m) 21s ago 15m 48.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 4a83b787c605 2024-09-17T12:30:47.018 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (2m) 2m ago 13m 24.7M 2048M 19.3.0-4937-g74984932 6c1fcab85602 dc48dfc9e40c 2024-09-17T12:30:47.018 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (3m) 21s ago 13m 43.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 aa5074acd22a 2024-09-17T12:30:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (9m) 21s ago 9m 15.0M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:30:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (9m) 2m ago 9m 15.4M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:30:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (2m) 21s ago 13m 45.3M 2737M 19.3.0-4937-g74984932 6c1fcab85602 d12e6f26f048 2024-09-17T12:30:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (92s) 21s ago 12m 53.0M 2737M 19.3.0-4937-g74984932 6c1fcab85602 cd2f6e2964dd 2024-09-17T12:30:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (59s) 21s ago 12m 42.0M 2737M 19.3.0-4937-g74984932 6c1fcab85602 99430f24b824 2024-09-17T12:30:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (26s) 21s ago 12m 11.5M 2737M 19.3.0-4937-g74984932 6c1fcab85602 1d4309778a0d 2024-09-17T12:30:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (11m) 2m ago 11m 55.7M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 afa20056b398 2024-09-17T12:30:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (11m) 2m ago 11m 51.2M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 f0dcabb8902f 2024-09-17T12:30:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (10m) 2m ago 10m 60.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:30:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (10m) 2m ago 10m 61.0M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:30:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (3m) 2m ago 10m 42.7M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:30:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (8m) 21s ago 8m 80.7M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:30:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (8m) 2m ago 8m 80.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:30:47.020 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (8m) 21s ago 8m 83.1M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:30:47.020 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (8m) 2m ago 8m 79.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:30:47.376 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:30:47.376 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:30:47.376 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:30:47.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:30:47.376 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:30:47.377 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:30:47.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:30:47.377 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:30:47.377 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4, 2024-09-17T12:30:47.377 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 4 2024-09-17T12:30:47.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:30:47.377 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:30:47.377 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:30:47.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:30:47.377 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:30:47.377 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8, 2024-09-17T12:30:47.378 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 9 2024-09-17T12:30:47.378 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:30:47.378 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:30:47.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:47 smithi016 ceph-mon[87889]: from='client.44306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:47.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:47 smithi016 ceph-mon[87889]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:47.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:47 smithi016 ceph-mon[87889]: from='client.44318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:47.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:47 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/983168045' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:47.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:47 smithi016 ceph-mon[89784]: from='client.44306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:47.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:47 smithi016 ceph-mon[89784]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:47.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:47 smithi016 ceph-mon[89784]: from='client.44318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:47.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:47 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/983168045' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:47.710 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:30:47.710 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:30:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:30:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:30:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:30:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-09-17T12:30:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:30:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:30:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "9/23 daemons upgraded", 2024-09-17T12:30:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T12:30:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:30:47.711 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:30:47.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:47 smithi196 ceph-mon[80339]: from='client.44306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:47.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:47 smithi196 ceph-mon[80339]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:47.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:47 smithi196 ceph-mon[80339]: from='client.44318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:47.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:47 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/983168045' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:30:48.063 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-09-17T12:30:48.751 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:48 smithi196 ceph-mon[80339]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:30:48.751 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:48 smithi196 ceph-mon[80339]: from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:48.751 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:48 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/1212822298' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:30:48.752 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:48 smithi196 systemd[1]: Stopping Ceph osd.4 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:30:48.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:48 smithi016 ceph-mon[87889]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:30:48.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:48 smithi016 ceph-mon[87889]: from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:48.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:48 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/1212822298' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:30:48.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:48 smithi016 ceph-mon[89784]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:30:48.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:48 smithi016 ceph-mon[89784]: from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:30:48.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:48 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/1212822298' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:30:49.026 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:48 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[32195]: 2024-09-17T12:30:48.748+0000 7fd459212640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:30:49.026 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:48 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[32195]: 2024-09-17T12:30:48.748+0000 7fd459212640 -1 osd.4 124 *** Got signal Terminated *** 2024-09-17T12:30:49.027 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:48 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[32195]: 2024-09-17T12:30:48.748+0000 7fd459212640 -1 osd.4 124 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:30:49.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:49 smithi196 ceph-mon[80339]: osd.4 marked itself down and dead 2024-09-17T12:30:49.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:30:49.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:49 smithi016 ceph-mon[87889]: osd.4 marked itself down and dead 2024-09-17T12:30:49.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:30:49.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:49 smithi016 ceph-mon[89784]: osd.4 marked itself down and dead 2024-09-17T12:30:49.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:30:50.276 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:49 smithi196 podman[84764]: 2024-09-17 12:30:49.990625296 +0000 UTC m=+1.342774440 container died afa20056b398a2f77ceb26bd2a50286a294869ba04ecfdf60710cd4f065fa9f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:30:50.668 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:50 smithi196 podman[84764]: 2024-09-17 12:30:50.382365906 +0000 UTC m=+1.734515040 container cleanup afa20056b398a2f77ceb26bd2a50286a294869ba04ecfdf60710cd4f065fa9f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:30:50.668 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:50 smithi196 bash[84764]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4 2024-09-17T12:30:50.669 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:50 smithi196 ceph-mon[80339]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:30:50.669 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:50 smithi196 ceph-mon[80339]: osdmap e125: 8 total, 7 up, 8 in 2024-09-17T12:30:50.669 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:50 smithi196 ceph-mon[80339]: pgmap v122: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:30:50.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:50 smithi016 ceph-mon[87889]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:30:50.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:50 smithi016 ceph-mon[87889]: osdmap e125: 8 total, 7 up, 8 in 2024-09-17T12:30:50.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:50 smithi016 ceph-mon[87889]: pgmap v122: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:30:50.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:50 smithi016 ceph-mon[89784]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:30:50.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:50 smithi016 ceph-mon[89784]: osdmap e125: 8 total, 7 up, 8 in 2024-09-17T12:30:50.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:50 smithi016 ceph-mon[89784]: pgmap v122: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:30:50.996 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:50 smithi196 podman[84791]: 2024-09-17 12:30:50.66748498 +0000 UTC m=+0.671224325 container remove afa20056b398a2f77ceb26bd2a50286a294869ba04ecfdf60710cd4f065fa9f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-17T12:30:51.277 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:50 smithi196 podman[84879]: 2024-09-17 12:30:50.898533941 +0000 UTC m=+0.016084542 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:30:51.277 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:51 smithi196 podman[84879]: 2024-09-17 12:30:51.009843072 +0000 UTC m=+0.127393657 container create b01b0dd9ae75ad4da3f60a853f7b6979e7b1db1ea758a31ed1b2d6a565cad1c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=main-7498493, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-17T12:30:51.277 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:51 smithi196 podman[84879]: 2024-09-17 12:30:51.220959114 +0000 UTC m=+0.338509702 container init b01b0dd9ae75ad4da3f60a853f7b6979e7b1db1ea758a31ed1b2d6a565cad1c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=main-7498493, GIT_BRANCH=HEAD, ceph=True) 2024-09-17T12:30:51.277 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:51 smithi196 podman[84879]: 2024-09-17 12:30:51.225049414 +0000 UTC m=+0.342600003 container start b01b0dd9ae75ad4da3f60a853f7b6979e7b1db1ea758a31ed1b2d6a565cad1c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=main-7498493, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:30:51.726 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:51 smithi196 ceph-mon[80339]: osdmap e126: 8 total, 7 up, 8 in 2024-09-17T12:30:51.726 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:51 smithi196 podman[84879]: 2024-09-17 12:30:51.320914173 +0000 UTC m=+0.438464770 container attach b01b0dd9ae75ad4da3f60a853f7b6979e7b1db1ea758a31ed1b2d6a565cad1c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-7498493, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-17T12:30:51.726 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:51 smithi196 podman[84879]: 2024-09-17 12:30:51.415140895 +0000 UTC m=+0.532691491 container died b01b0dd9ae75ad4da3f60a853f7b6979e7b1db1ea758a31ed1b2d6a565cad1c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-7498493, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:30:51.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:51 smithi016 ceph-mon[87889]: osdmap e126: 8 total, 7 up, 8 in 2024-09-17T12:30:51.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:51 smithi016 ceph-mon[89784]: osdmap e126: 8 total, 7 up, 8 in 2024-09-17T12:30:52.503 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 podman[84903]: 2024-09-17 12:30:52.067688271 +0000 UTC m=+0.646643586 container remove b01b0dd9ae75ad4da3f60a853f7b6979e7b1db1ea758a31ed1b2d6a565cad1c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-7498493, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-09-17T12:30:52.504 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.4.service: Deactivated successfully. 2024-09-17T12:30:52.504 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 systemd[1]: Stopped Ceph osd.4 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:30:52.504 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.4.service: Consumed 7.711s CPU time. 2024-09-17T12:30:52.504 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 systemd[1]: Starting Ceph osd.4 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:30:52.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:52 smithi196 ceph-mon[80339]: pgmap v124: 161 pgs: 10 peering, 20 stale+active+clean, 131 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T12:30:52.777 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 podman[85001]: 2024-09-17 12:30:52.502308585 +0000 UTC m=+0.112680242 container create d76c465f38168777e06bde60c5ba402cb86a73aefab50b65e9165ee02ea48cd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=main-7498493, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-17T12:30:52.777 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 podman[85001]: 2024-09-17 12:30:52.406318459 +0000 UTC m=+0.016690120 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:30:52.777 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 podman[85001]: 2024-09-17 12:30:52.721547471 +0000 UTC m=+0.331919138 container init d76c465f38168777e06bde60c5ba402cb86a73aefab50b65e9165ee02ea48cd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-7498493, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:30:52.777 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 podman[85001]: 2024-09-17 12:30:52.725031307 +0000 UTC m=+0.335402965 container start d76c465f38168777e06bde60c5ba402cb86a73aefab50b65e9165ee02ea48cd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate, ceph=True, RELEASE=main-7498493, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-17T12:30:52.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:52 smithi016 ceph-mon[87889]: pgmap v124: 161 pgs: 10 peering, 20 stale+active+clean, 131 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T12:30:52.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:52 smithi016 ceph-mon[89784]: pgmap v124: 161 pgs: 10 peering, 20 stale+active+clean, 131 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T12:30:53.276 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 podman[85001]: 2024-09-17 12:30:52.813419088 +0000 UTC m=+0.423790752 container attach d76c465f38168777e06bde60c5ba402cb86a73aefab50b65e9165ee02ea48cd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-7498493, org.label-schema.schema-version=1.0) 2024-09-17T12:30:53.277 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate[85015]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:30:53.277 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 bash[85001]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:30:53.277 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate[85015]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:30:53.277 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:52 smithi196 bash[85001]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:30:53.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:53 smithi196 ceph-mon[80339]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T12:30:53.617 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate[85015]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T12:30:53.617 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 bash[85001]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T12:30:53.618 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate[85015]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T12:30:53.618 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 bash[85001]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T12:30:53.618 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate[85015]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T12:30:53.618 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 bash[85001]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T12:30:53.934 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate[85015]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T12:30:53.935 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 bash[85001]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T12:30:53.935 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate[85015]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-17T12:30:53.935 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 bash[85001]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-17T12:30:53.935 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate[85015]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T12:30:53.935 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 bash[85001]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T12:30:53.935 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate[85015]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-17T12:30:53.935 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 bash[85001]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-17T12:30:53.935 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:53 smithi196 podman[85001]: 2024-09-17 12:30:53.644581791 +0000 UTC m=+1.254953450 container died d76c465f38168777e06bde60c5ba402cb86a73aefab50b65e9165ee02ea48cd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-7498493, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-17T12:30:53.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:53 smithi016 ceph-mon[87889]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T12:30:53.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:53 smithi016 ceph-mon[89784]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T12:30:54.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:54 smithi196 ceph-mon[80339]: pgmap v125: 161 pgs: 16 active+undersized, 10 peering, 9 stale+active+clean, 10 active+undersized+degraded, 116 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 36/633 objects degraded (5.687%) 2024-09-17T12:30:54.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:54 smithi196 ceph-mon[80339]: Health check failed: Degraded data redundancy: 36/633 objects degraded (5.687%), 10 pgs degraded (PG_DEGRADED) 2024-09-17T12:30:54.650 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:54 smithi196 podman[85146]: 2024-09-17 12:30:54.30098402 +0000 UTC m=+0.645307866 container remove d76c465f38168777e06bde60c5ba402cb86a73aefab50b65e9165ee02ea48cd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-7498493, org.label-schema.build-date=20240731) 2024-09-17T12:30:54.650 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:54 smithi196 podman[85207]: 2024-09-17 12:30:54.416148784 +0000 UTC m=+0.016566025 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:30:54.650 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:54 smithi196 podman[85207]: 2024-09-17 12:30:54.526877593 +0000 UTC m=+0.127294857 container create 0c739e0cd187a4a6e87714eaa5ab8ddefdc5669f29997a2b89a66fafb2ea866c (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4, RELEASE=main-7498493, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-09-17T12:30:54.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:54 smithi016 ceph-mon[87889]: pgmap v125: 161 pgs: 16 active+undersized, 10 peering, 9 stale+active+clean, 10 active+undersized+degraded, 116 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 36/633 objects degraded (5.687%) 2024-09-17T12:30:54.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:54 smithi016 ceph-mon[87889]: Health check failed: Degraded data redundancy: 36/633 objects degraded (5.687%), 10 pgs degraded (PG_DEGRADED) 2024-09-17T12:30:54.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:54 smithi016 ceph-mon[89784]: pgmap v125: 161 pgs: 16 active+undersized, 10 peering, 9 stale+active+clean, 10 active+undersized+degraded, 116 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 36/633 objects degraded (5.687%) 2024-09-17T12:30:54.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:54 smithi016 ceph-mon[89784]: Health check failed: Degraded data redundancy: 36/633 objects degraded (5.687%), 10 pgs degraded (PG_DEGRADED) 2024-09-17T12:30:55.027 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:54 smithi196 podman[85207]: 2024-09-17 12:30:54.772106414 +0000 UTC m=+0.372523639 container init 0c739e0cd187a4a6e87714eaa5ab8ddefdc5669f29997a2b89a66fafb2ea866c (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-7498493, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-09-17T12:30:55.027 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:54 smithi196 podman[85207]: 2024-09-17 12:30:54.775326778 +0000 UTC m=+0.375744000 container start 0c739e0cd187a4a6e87714eaa5ab8ddefdc5669f29997a2b89a66fafb2ea866c (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-7498493, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:30:55.027 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:54 smithi196 ceph-osd[85226]: -- 172.21.15.196:0/457975607 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c4b1699a40 con 0x55c4b2468400 2024-09-17T12:30:55.027 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:54 smithi196 bash[85207]: 0c739e0cd187a4a6e87714eaa5ab8ddefdc5669f29997a2b89a66fafb2ea866c 2024-09-17T12:30:55.027 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:54 smithi196 systemd[1]: Started Ceph osd.4 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:30:56.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:55 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:56.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:55 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:56.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:55 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:30:56.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:55 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:56.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:55 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:56.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:55 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:30:56.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:55 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:55 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:30:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:55 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:30:56.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:30:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:30:56] "GET /metrics HTTP/1.1" 200 38629 "" "Prometheus/2.43.0" 2024-09-17T12:30:56.777 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:30:56.650+0000 7f16544e2640 -1 Falling back to public interface 2024-09-17T12:30:57.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:56 smithi016 ceph-mon[87889]: pgmap v126: 161 pgs: 40 active+undersized, 28 active+undersized+degraded, 93 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 107/633 objects degraded (16.904%) 2024-09-17T12:30:57.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:56 smithi016 ceph-mon[87889]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-17T12:30:57.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:56 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:57.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:56 smithi016 ceph-mon[89784]: pgmap v126: 161 pgs: 40 active+undersized, 28 active+undersized+degraded, 93 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 107/633 objects degraded (16.904%) 2024-09-17T12:30:57.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:56 smithi016 ceph-mon[89784]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-17T12:30:57.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:56 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:57.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:56 smithi196 ceph-mon[80339]: pgmap v126: 161 pgs: 40 active+undersized, 28 active+undersized+degraded, 93 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 107/633 objects degraded (16.904%) 2024-09-17T12:30:57.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:56 smithi196 ceph-mon[80339]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-17T12:30:57.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:56 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:30:59.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:30:58 smithi016 ceph-mon[87889]: pgmap v127: 161 pgs: 40 active+undersized, 28 active+undersized+degraded, 93 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 107/633 objects degraded (16.904%) 2024-09-17T12:30:59.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:30:58 smithi016 ceph-mon[89784]: pgmap v127: 161 pgs: 40 active+undersized, 28 active+undersized+degraded, 93 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 107/633 objects degraded (16.904%) 2024-09-17T12:30:59.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:30:58 smithi196 ceph-mon[80339]: pgmap v127: 161 pgs: 40 active+undersized, 28 active+undersized+degraded, 93 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 107/633 objects degraded (16.904%) 2024-09-17T12:31:00.027 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:30:59 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:30:59.979+0000 7f16544e2640 -1 osd.4 124 log_to_monitors true 2024-09-17T12:31:00.366 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:00 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:00.366 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:00 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:00.366 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:00 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:00.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:00 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:00.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:00 smithi196 ceph-mon[80339]: from='osd.4 [v2:172.21.15.196:6800/1183411641,v1:172.21.15.196:6801/1183411641]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-17T12:31:00.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:00 smithi196 ceph-mon[80339]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-17T12:31:00.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:00 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:00.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:00 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:00.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:00 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:00.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:00 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:00.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:00 smithi016 ceph-mon[87889]: from='osd.4 [v2:172.21.15.196:6800/1183411641,v1:172.21.15.196:6801/1183411641]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-17T12:31:00.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:00 smithi016 ceph-mon[87889]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-17T12:31:00.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:00 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:00.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:00 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:00.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:00 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:00.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:00 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:00.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:00 smithi016 ceph-mon[89784]: from='osd.4 [v2:172.21.15.196:6800/1183411641,v1:172.21.15.196:6801/1183411641]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-17T12:31:00.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:00 smithi016 ceph-mon[89784]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-17T12:31:00.928 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:31:00 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:31:00.656+0000 7f164c28c640 -1 osd.4 124 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:31:01.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:01 smithi196 ceph-mon[80339]: pgmap v128: 161 pgs: 40 active+undersized, 28 active+undersized+degraded, 93 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s; 107/633 objects degraded (16.904%) 2024-09-17T12:31:01.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:01 smithi196 ceph-mon[80339]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T12:31:01.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:01 smithi196 ceph-mon[80339]: from='osd.4 [v2:172.21.15.196:6800/1183411641,v1:172.21.15.196:6801/1183411641]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:31:01.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:01 smithi196 ceph-mon[80339]: osdmap e127: 8 total, 7 up, 8 in 2024-09-17T12:31:01.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:01 smithi196 ceph-mon[80339]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:31:01.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:01 smithi016 ceph-mon[87889]: pgmap v128: 161 pgs: 40 active+undersized, 28 active+undersized+degraded, 93 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s; 107/633 objects degraded (16.904%) 2024-09-17T12:31:01.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:01 smithi016 ceph-mon[87889]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T12:31:01.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:01 smithi016 ceph-mon[87889]: from='osd.4 [v2:172.21.15.196:6800/1183411641,v1:172.21.15.196:6801/1183411641]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:31:01.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:01 smithi016 ceph-mon[87889]: osdmap e127: 8 total, 7 up, 8 in 2024-09-17T12:31:01.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:01 smithi016 ceph-mon[87889]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:31:01.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:01 smithi016 ceph-mon[89784]: pgmap v128: 161 pgs: 40 active+undersized, 28 active+undersized+degraded, 93 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s; 107/633 objects degraded (16.904%) 2024-09-17T12:31:01.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:01 smithi016 ceph-mon[89784]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T12:31:01.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:01 smithi016 ceph-mon[89784]: from='osd.4 [v2:172.21.15.196:6800/1183411641,v1:172.21.15.196:6801/1183411641]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:31:01.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:01 smithi016 ceph-mon[89784]: osdmap e127: 8 total, 7 up, 8 in 2024-09-17T12:31:01.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:01 smithi016 ceph-mon[89784]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:31:02.219 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:02 smithi196 ceph-mon[80339]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:31:02.220 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:02 smithi196 ceph-mon[80339]: osd.4 [v2:172.21.15.196:6800/1183411641,v1:172.21.15.196:6801/1183411641] boot 2024-09-17T12:31:02.220 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:02 smithi196 ceph-mon[80339]: osdmap e128: 8 total, 8 up, 8 in 2024-09-17T12:31:02.220 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:02 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T12:31:02.220 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:02 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 107/633 objects degraded (16.904%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:02.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:02 smithi016 ceph-mon[87889]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:31:02.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:02 smithi016 ceph-mon[87889]: osd.4 [v2:172.21.15.196:6800/1183411641,v1:172.21.15.196:6801/1183411641] boot 2024-09-17T12:31:02.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:02 smithi016 ceph-mon[87889]: osdmap e128: 8 total, 8 up, 8 in 2024-09-17T12:31:02.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:02 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T12:31:02.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:02 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 107/633 objects degraded (16.904%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:02.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:02 smithi016 ceph-mon[89784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:31:02.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:02 smithi016 ceph-mon[89784]: osd.4 [v2:172.21.15.196:6800/1183411641,v1:172.21.15.196:6801/1183411641] boot 2024-09-17T12:31:02.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:02 smithi016 ceph-mon[89784]: osdmap e128: 8 total, 8 up, 8 in 2024-09-17T12:31:02.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:02 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T12:31:02.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:02 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 107/633 objects degraded (16.904%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:02.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:31:02 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:31:02.682+0000 7f9d37c73640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T12:31:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[87889]: pgmap v130: 161 pgs: 40 active+undersized, 28 active+undersized+degraded, 93 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 107/633 objects degraded (16.904%) 2024-09-17T12:31:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:03.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:31:03.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:31:03.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:03.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[87889]: osdmap e129: 8 total, 8 up, 8 in 2024-09-17T12:31:03.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:31:03.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:03.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:03.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:03.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-17T12:31:03.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[89784]: pgmap v130: 161 pgs: 40 active+undersized, 28 active+undersized+degraded, 93 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 107/633 objects degraded (16.904%) 2024-09-17T12:31:03.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:03.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:03.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:31:03.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:31:03.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:03.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[89784]: osdmap e129: 8 total, 8 up, 8 in 2024-09-17T12:31:03.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:31:03.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:03.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:03.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:03.444 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:03 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-17T12:31:03.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:03 smithi196 ceph-mon[80339]: pgmap v130: 161 pgs: 40 active+undersized, 28 active+undersized+degraded, 93 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 107/633 objects degraded (16.904%) 2024-09-17T12:31:03.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:03 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:03.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:03 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:03.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:03 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:31:03.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:03 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:31:03.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:03 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:03.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:03 smithi196 ceph-mon[80339]: osdmap e129: 8 total, 8 up, 8 in 2024-09-17T12:31:03.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:03 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:31:03.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:03 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:03 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:03 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:03.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:03 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-17T12:31:04.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:04 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T12:31:04.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:04 smithi016 ceph-mon[87889]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T12:31:04.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:04 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:04.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:04 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:31:04.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:04 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T12:31:04.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:04 smithi016 ceph-mon[89784]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T12:31:04.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:04 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:04.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:04 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:31:04.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:04 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T12:31:04.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:04 smithi196 ceph-mon[80339]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T12:31:04.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:04 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:04.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:04 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:31:05.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:05 smithi016 ceph-mon[87889]: pgmap v133: 161 pgs: 11 peering, 30 active+undersized, 23 active+undersized+degraded, 97 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 87/633 objects degraded (13.744%) 2024-09-17T12:31:05.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:05 smithi016 ceph-mon[89784]: pgmap v133: 161 pgs: 11 peering, 30 active+undersized, 23 active+undersized+degraded, 97 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 87/633 objects degraded (13.744%) 2024-09-17T12:31:05.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:05 smithi196 ceph-mon[80339]: pgmap v133: 161 pgs: 11 peering, 30 active+undersized, 23 active+undersized+degraded, 97 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 87/633 objects degraded (13.744%) 2024-09-17T12:31:06.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:31:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:31:06] "GET /metrics HTTP/1.1" 200 38480 "" "Prometheus/2.43.0" 2024-09-17T12:31:07.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:07 smithi196 ceph-mon[80339]: pgmap v134: 161 pgs: 11 peering, 13 active+undersized, 10 active+undersized+degraded, 127 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 28/633 objects degraded (4.423%) 2024-09-17T12:31:07.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:07 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:07.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:07 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 28/633 objects degraded (4.423%), 10 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:07.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:07 smithi016 ceph-mon[89784]: pgmap v134: 161 pgs: 11 peering, 13 active+undersized, 10 active+undersized+degraded, 127 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 28/633 objects degraded (4.423%) 2024-09-17T12:31:07.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:07 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:07.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:07 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 28/633 objects degraded (4.423%), 10 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:07.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:07 smithi016 ceph-mon[87889]: pgmap v134: 161 pgs: 11 peering, 13 active+undersized, 10 active+undersized+degraded, 127 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 28/633 objects degraded (4.423%) 2024-09-17T12:31:07.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:07 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:07.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:07 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 28/633 objects degraded (4.423%), 10 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:08.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:08 smithi016 ceph-mon[89784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/633 objects degraded (4.423%), 10 pgs degraded) 2024-09-17T12:31:08.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:08 smithi016 ceph-mon[89784]: Cluster is now healthy 2024-09-17T12:31:08.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:08 smithi016 ceph-mon[87889]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/633 objects degraded (4.423%), 10 pgs degraded) 2024-09-17T12:31:08.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:08 smithi016 ceph-mon[87889]: Cluster is now healthy 2024-09-17T12:31:08.506 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:08 smithi196 ceph-mon[80339]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/633 objects degraded (4.423%), 10 pgs degraded) 2024-09-17T12:31:08.506 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:08 smithi196 ceph-mon[80339]: Cluster is now healthy 2024-09-17T12:31:09.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:09 smithi016 ceph-mon[89784]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:09.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:09 smithi016 ceph-mon[87889]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:09.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:09 smithi196 ceph-mon[80339]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:11.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:11 smithi016 ceph-mon[87889]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:11.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:11 smithi016 ceph-mon[89784]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:11.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:11 smithi196 ceph-mon[80339]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:13.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:13 smithi016 ceph-mon[87889]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:13.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:13 smithi016 ceph-mon[89784]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:13.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:13 smithi196 ceph-mon[80339]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:15.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:15 smithi196 ceph-mon[80339]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 940 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:15.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:15.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:15 smithi016 ceph-mon[87889]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 940 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:15.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:15.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:15 smithi016 ceph-mon[89784]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 940 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:15.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:16.599 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:31:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:31:16] "GET /metrics HTTP/1.1" 200 38480 "" "Prometheus/2.43.0" 2024-09-17T12:31:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:17 smithi196 ceph-mon[80339]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:17 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:17.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:17 smithi016 ceph-mon[87889]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:17.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:17 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:17.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:17 smithi016 ceph-mon[89784]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:17.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:17 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:18.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:18 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-17T12:31:18.379 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:31:18.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:18 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-17T12:31:18.379 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:18 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-17T12:31:18.934 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:31:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=cleanup t=2024-09-17T12:31:18.662938748Z level=info msg="Completed cleanup jobs" duration=12.895098ms 2024-09-17T12:31:18.967 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:31:18.967 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (9m) 53s ago 10m 25.0M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:31:18.967 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (10m) 19s ago 10m 86.0M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:31:18.967 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (4m) 53s ago 8m 60.9M - 3.8 980a0bfcd684 843db1407344 2024-09-17T12:31:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (4m) 19s ago 14m 454M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:31:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (4m) 53s ago 15m 519M - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:31:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (3m) 53s ago 15m 48.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 4a83b787c605 2024-09-17T12:31:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (3m) 19s ago 14m 40.3M 2048M 19.3.0-4937-g74984932 6c1fcab85602 dc48dfc9e40c 2024-09-17T12:31:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (3m) 53s ago 14m 43.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 aa5074acd22a 2024-09-17T12:31:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (9m) 53s ago 9m 15.0M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:31:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (9m) 19s ago 9m 15.8M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:31:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (2m) 53s ago 13m 45.3M 2737M 19.3.0-4937-g74984932 6c1fcab85602 d12e6f26f048 2024-09-17T12:31:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (2m) 53s ago 13m 53.0M 2737M 19.3.0-4937-g74984932 6c1fcab85602 cd2f6e2964dd 2024-09-17T12:31:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (91s) 53s ago 13m 42.0M 2737M 19.3.0-4937-g74984932 6c1fcab85602 99430f24b824 2024-09-17T12:31:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (58s) 53s ago 12m 11.5M 2737M 19.3.0-4937-g74984932 6c1fcab85602 1d4309778a0d 2024-09-17T12:31:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (24s) 19s ago 12m 11.6M 2992M 19.3.0-4937-g74984932 6c1fcab85602 0c739e0cd187 2024-09-17T12:31:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (11m) 19s ago 11m 55.1M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 f0dcabb8902f 2024-09-17T12:31:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (11m) 19s ago 11m 63.3M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:31:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (11m) 19s ago 11m 66.4M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:31:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (4m) 19s ago 10m 52.0M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:31:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (8m) 53s ago 8m 80.7M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:31:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (8m) 19s ago 8m 81.5M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:31:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (8m) 53s ago 8m 83.1M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:31:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (8m) 19s ago 8m 81.0M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:31:19.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:19 smithi196 ceph-mon[80339]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:19.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:19 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T12:31:19.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:19 smithi196 ceph-mon[80339]: Upgrade: osd.5 is safe to restart 2024-09-17T12:31:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:19 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:19 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:31:19.320 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:31:19.320 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:31:19.320 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:31:19.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:31:19.320 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:31:19.320 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:31:19.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:31:19.321 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:31:19.321 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3, 2024-09-17T12:31:19.321 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 5 2024-09-17T12:31:19.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:31:19.321 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:31:19.321 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:31:19.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:31:19.321 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:31:19.321 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 7, 2024-09-17T12:31:19.321 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 10 2024-09-17T12:31:19.321 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:31:19.321 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:31:19.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:19 smithi016 ceph-mon[87889]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:19.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:19 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T12:31:19.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:19 smithi016 ceph-mon[87889]: Upgrade: osd.5 is safe to restart 2024-09-17T12:31:19.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:19 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:19.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:19 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:31:19.322 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:19 smithi016 ceph-mon[89784]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T12:31:19.322 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:19 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T12:31:19.322 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:19 smithi016 ceph-mon[89784]: Upgrade: osd.5 is safe to restart 2024-09-17T12:31:19.323 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:19 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:19.323 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:19 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:31:19.620 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:31:19.620 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:31:19.620 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:31:19.620 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:31:19.620 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:31:19.620 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-09-17T12:31:19.620 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:31:19.620 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:31:19.620 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "10/23 daemons upgraded", 2024-09-17T12:31:19.620 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T12:31:19.620 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:31:19.621 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:31:19.973 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-09-17T12:31:20.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:20 smithi196 ceph-mon[80339]: from='client.44342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:20 smithi196 ceph-mon[80339]: from='client.44348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:20 smithi196 ceph-mon[80339]: from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:20 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/1286401922' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:20 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:20 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-17T12:31:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:20 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:31:20.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:20 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/58117267' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:31:20.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[87889]: from='client.44342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:20.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[87889]: from='client.44348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:20.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[87889]: from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:20.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/1286401922' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:20.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:20.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-17T12:31:20.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:31:20.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/58117267' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:31:20.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[89784]: from='client.44342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:20.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[89784]: from='client.44348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:20.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[89784]: from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:20.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/1286401922' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:20.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:20.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-17T12:31:20.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:31:20.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:20 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/58117267' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:31:21.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:21 smithi196 ceph-mon[80339]: Upgrade: Updating osd.5 2024-09-17T12:31:21.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:21 smithi196 ceph-mon[80339]: Deploying daemon osd.5 on smithi196 2024-09-17T12:31:21.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:21 smithi196 ceph-mon[80339]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:21.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:21 smithi196 ceph-mon[80339]: from='client.34431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:21.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:21 smithi016 ceph-mon[87889]: Upgrade: Updating osd.5 2024-09-17T12:31:21.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:21 smithi016 ceph-mon[87889]: Deploying daemon osd.5 on smithi196 2024-09-17T12:31:21.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:21 smithi016 ceph-mon[87889]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:21.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:21 smithi016 ceph-mon[87889]: from='client.34431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:21.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:21 smithi016 ceph-mon[89784]: Upgrade: Updating osd.5 2024-09-17T12:31:21.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:21 smithi016 ceph-mon[89784]: Deploying daemon osd.5 on smithi196 2024-09-17T12:31:21.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:21 smithi016 ceph-mon[89784]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:21.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:21 smithi016 ceph-mon[89784]: from='client.34431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:22.527 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:22 smithi196 systemd[1]: Stopping Ceph osd.5 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:31:23.026 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:22 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[37768]: 2024-09-17T12:31:22.560+0000 7f323c642640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:31:23.027 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:22 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[37768]: 2024-09-17T12:31:22.560+0000 7f323c642640 -1 osd.5 129 *** Got signal Terminated *** 2024-09-17T12:31:23.027 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:22 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[37768]: 2024-09-17T12:31:22.560+0000 7f323c642640 -1 osd.5 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:31:23.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:23 smithi016 ceph-mon[87889]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:23.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:23 smithi016 ceph-mon[87889]: osd.5 marked itself down and dead 2024-09-17T12:31:23.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:23 smithi016 ceph-mon[89784]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:23.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:23 smithi016 ceph-mon[89784]: osd.5 marked itself down and dead 2024-09-17T12:31:23.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:23 smithi196 ceph-mon[80339]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:23.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:23 smithi196 ceph-mon[80339]: osd.5 marked itself down and dead 2024-09-17T12:31:23.763 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:23 smithi196 podman[89106]: 2024-09-17 12:31:23.471041802 +0000 UTC m=+1.039498365 container died f0dcabb8902f3c45bd03742b40ca85780b346bffc9e951f0a0514ae41226b64a (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:31:24.026 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:23 smithi196 podman[89106]: 2024-09-17 12:31:23.868460757 +0000 UTC m=+1.436917319 container cleanup f0dcabb8902f3c45bd03742b40ca85780b346bffc9e951f0a0514ae41226b64a (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-17T12:31:24.026 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:23 smithi196 bash[89106]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5 2024-09-17T12:31:24.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:24 smithi016 ceph-mon[87889]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:31:24.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:24 smithi016 ceph-mon[87889]: osdmap e130: 8 total, 7 up, 8 in 2024-09-17T12:31:24.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:24 smithi016 ceph-mon[89784]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:31:24.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:24 smithi016 ceph-mon[89784]: osdmap e130: 8 total, 7 up, 8 in 2024-09-17T12:31:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:24 smithi196 ceph-mon[80339]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:31:24.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:24 smithi196 ceph-mon[80339]: osdmap e130: 8 total, 7 up, 8 in 2024-09-17T12:31:24.527 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:24 smithi196 podman[89122]: 2024-09-17 12:31:24.178304771 +0000 UTC m=+0.701034131 container remove f0dcabb8902f3c45bd03742b40ca85780b346bffc9e951f0a0514ae41226b64a (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-17T12:31:24.849 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:24 smithi196 podman[89208]: 2024-09-17 12:31:24.432455816 +0000 UTC m=+0.015955363 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:31:24.849 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:24 smithi196 podman[89208]: 2024-09-17 12:31:24.545910946 +0000 UTC m=+0.129410513 container create 5ecf138463abbb54748b1ed8ab2afb6ea7cd07d6493fa36b27a2a58b70b189b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=main-7498493, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD) 2024-09-17T12:31:24.850 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:24 smithi196 podman[89208]: 2024-09-17 12:31:24.756656691 +0000 UTC m=+0.340156231 container init 5ecf138463abbb54748b1ed8ab2afb6ea7cd07d6493fa36b27a2a58b70b189b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-7498493, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:31:24.850 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:24 smithi196 podman[89208]: 2024-09-17 12:31:24.76010477 +0000 UTC m=+0.343604314 container start 5ecf138463abbb54748b1ed8ab2afb6ea7cd07d6493fa36b27a2a58b70b189b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:31:24.850 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:24 smithi196 podman[89208]: 2024-09-17 12:31:24.848350741 +0000 UTC m=+0.431850276 container attach 5ecf138463abbb54748b1ed8ab2afb6ea7cd07d6493fa36b27a2a58b70b189b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=main-7498493, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-17T12:31:25.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:25 smithi196 ceph-mon[80339]: pgmap v144: 161 pgs: 6 peering, 20 stale+active+clean, 135 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:31:25.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:25 smithi196 ceph-mon[80339]: osdmap e131: 8 total, 7 up, 8 in 2024-09-17T12:31:25.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:25 smithi196 ceph-mon[80339]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-17T12:31:25.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:24 smithi196 podman[89208]: 2024-09-17 12:31:24.953226617 +0000 UTC m=+0.536726178 container died 5ecf138463abbb54748b1ed8ab2afb6ea7cd07d6493fa36b27a2a58b70b189b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-17T12:31:25.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:25 smithi016 ceph-mon[87889]: pgmap v144: 161 pgs: 6 peering, 20 stale+active+clean, 135 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:31:25.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:25 smithi016 ceph-mon[87889]: osdmap e131: 8 total, 7 up, 8 in 2024-09-17T12:31:25.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:25 smithi016 ceph-mon[87889]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-17T12:31:25.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:25 smithi016 ceph-mon[89784]: pgmap v144: 161 pgs: 6 peering, 20 stale+active+clean, 135 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:31:25.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:25 smithi016 ceph-mon[89784]: osdmap e131: 8 total, 7 up, 8 in 2024-09-17T12:31:25.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:25 smithi016 ceph-mon[89784]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-17T12:31:26.026 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:25 smithi196 podman[89232]: 2024-09-17 12:31:25.694106695 +0000 UTC m=+0.731542300 container remove 5ecf138463abbb54748b1ed8ab2afb6ea7cd07d6493fa36b27a2a58b70b189b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-7498493) 2024-09-17T12:31:26.027 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:25 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.5.service: Deactivated successfully. 2024-09-17T12:31:26.027 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:25 smithi196 systemd[1]: Stopped Ceph osd.5 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:31:26.027 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:25 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.5.service: Consumed 7.766s CPU time. 2024-09-17T12:31:26.027 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:25 smithi196 systemd[1]: Starting Ceph osd.5 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:31:26.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:26 smithi196 ceph-mon[80339]: Health check failed: Degraded data redundancy: 26/633 objects degraded (4.107%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:26.374 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:26 smithi196 podman[89331]: 2024-09-17 12:31:26.018941003 +0000 UTC m=+0.015653942 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:31:26.374 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:26 smithi196 podman[89331]: 2024-09-17 12:31:26.145470504 +0000 UTC m=+0.142183423 container create 17a6bd0fd71c97a9f12059b9c28dfa711b044ebebffba1c414df27f11d57de64 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-7498493, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-17T12:31:26.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:26 smithi016 ceph-mon[87889]: Health check failed: Degraded data redundancy: 26/633 objects degraded (4.107%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:26.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:26 smithi016 ceph-mon[89784]: Health check failed: Degraded data redundancy: 26/633 objects degraded (4.107%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:26.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:31:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:31:26] "GET /metrics HTTP/1.1" 200 38623 "" "Prometheus/2.43.0" 2024-09-17T12:31:26.777 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:26 smithi196 podman[89331]: 2024-09-17 12:31:26.372638357 +0000 UTC m=+0.369351294 container init 17a6bd0fd71c97a9f12059b9c28dfa711b044ebebffba1c414df27f11d57de64 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-7498493, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-17T12:31:26.777 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:26 smithi196 podman[89331]: 2024-09-17 12:31:26.376048842 +0000 UTC m=+0.372761767 container start 17a6bd0fd71c97a9f12059b9c28dfa711b044ebebffba1c414df27f11d57de64 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=main-7498493, CEPH_POINT_RELEASE=) 2024-09-17T12:31:26.777 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:26 smithi196 podman[89331]: 2024-09-17 12:31:26.489645047 +0000 UTC m=+0.486357971 container attach 17a6bd0fd71c97a9f12059b9c28dfa711b044ebebffba1c414df27f11d57de64 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-7498493, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:31:26.777 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:26 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate[89346]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:31:26.777 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:26 smithi196 bash[89331]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:31:26.777 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:26 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate[89346]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:31:26.777 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:26 smithi196 bash[89331]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:31:27.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:27 smithi196 ceph-mon[80339]: pgmap v146: 161 pgs: 22 active+undersized, 6 peering, 10 stale+active+clean, 6 active+undersized+degraded, 117 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 26/633 objects degraded (4.107%) 2024-09-17T12:31:27.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:27 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:27.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:26 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate[89346]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T12:31:27.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:26 smithi196 bash[89331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T12:31:27.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:26 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate[89346]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T12:31:27.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:26 smithi196 bash[89331]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T12:31:27.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:27 smithi016 ceph-mon[87889]: pgmap v146: 161 pgs: 22 active+undersized, 6 peering, 10 stale+active+clean, 6 active+undersized+degraded, 117 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 26/633 objects degraded (4.107%) 2024-09-17T12:31:27.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:27 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:27.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:27 smithi016 ceph-mon[89784]: pgmap v146: 161 pgs: 22 active+undersized, 6 peering, 10 stale+active+clean, 6 active+undersized+degraded, 117 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 26/633 objects degraded (4.107%) 2024-09-17T12:31:27.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:27 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:27.610 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:27 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate[89346]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T12:31:27.611 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:27 smithi196 bash[89331]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T12:31:27.611 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:27 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate[89346]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T12:31:27.611 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:27 smithi196 bash[89331]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T12:31:27.611 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:27 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate[89346]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-17T12:31:27.611 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:27 smithi196 bash[89331]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-17T12:31:27.611 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:27 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate[89346]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T12:31:27.611 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:27 smithi196 bash[89331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T12:31:27.611 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:27 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate[89346]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-17T12:31:27.611 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:27 smithi196 bash[89331]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-17T12:31:27.611 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:27 smithi196 podman[89331]: 2024-09-17 12:31:27.311970068 +0000 UTC m=+1.308682994 container died 17a6bd0fd71c97a9f12059b9c28dfa711b044ebebffba1c414df27f11d57de64 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-7498493, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-17T12:31:28.276 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:28 smithi196 podman[89473]: 2024-09-17 12:31:28.001988399 +0000 UTC m=+0.683940442 container remove 17a6bd0fd71c97a9f12059b9c28dfa711b044ebebffba1c414df27f11d57de64 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, io.buildah.version=1.37.2) 2024-09-17T12:31:28.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:28 smithi196 podman[89531]: 2024-09-17 12:31:28.202648466 +0000 UTC m=+0.105478956 container create 71f93c731227c5eec5e895f24bde851f003f3a358eaa75e9892987fe597964af (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-7498493, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:31:28.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:28 smithi196 podman[89531]: 2024-09-17 12:31:28.11356998 +0000 UTC m=+0.016400478 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:31:28.777 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:28 smithi196 podman[89531]: 2024-09-17 12:31:28.421838019 +0000 UTC m=+0.324668516 container init 71f93c731227c5eec5e895f24bde851f003f3a358eaa75e9892987fe597964af (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-7498493, ceph=True) 2024-09-17T12:31:28.777 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:28 smithi196 podman[89531]: 2024-09-17 12:31:28.425131644 +0000 UTC m=+0.327962140 container start 71f93c731227c5eec5e895f24bde851f003f3a358eaa75e9892987fe597964af (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T12:31:28.777 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:28 smithi196 ceph-osd[89550]: -- 172.21.15.196:0/766182098 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55743e745a40 con 0x55743e723c00 2024-09-17T12:31:28.777 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:28 smithi196 bash[89531]: 71f93c731227c5eec5e895f24bde851f003f3a358eaa75e9892987fe597964af 2024-09-17T12:31:28.777 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:28 smithi196 systemd[1]: Started Ceph osd.5 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:31:29.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:29 smithi196 ceph-mon[80339]: pgmap v147: 161 pgs: 34 active+undersized, 6 peering, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 71/633 objects degraded (11.216%) 2024-09-17T12:31:29.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:29 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:29.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:29 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:29.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:29 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:31:29.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:29 smithi016 ceph-mon[87889]: pgmap v147: 161 pgs: 34 active+undersized, 6 peering, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 71/633 objects degraded (11.216%) 2024-09-17T12:31:29.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:29 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:29.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:29 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:29.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:29 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:31:29.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:29 smithi016 ceph-mon[89784]: pgmap v147: 161 pgs: 34 active+undersized, 6 peering, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 71/633 objects degraded (11.216%) 2024-09-17T12:31:29.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:29 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:29.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:29 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:29.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:29 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:31:30.143 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:29 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:31:29.829+0000 7fb324e49640 -1 Falling back to public interface 2024-09-17T12:31:30.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:30 smithi016 ceph-mon[87889]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-17T12:31:30.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:30 smithi016 ceph-mon[89784]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-17T12:31:30.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:30 smithi196 ceph-mon[80339]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-17T12:31:31.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:31 smithi016 ceph-mon[87889]: pgmap v148: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 71/633 objects degraded (11.216%) 2024-09-17T12:31:31.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:31 smithi016 ceph-mon[89784]: pgmap v148: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 71/633 objects degraded (11.216%) 2024-09-17T12:31:31.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:31 smithi196 ceph-mon[80339]: pgmap v148: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 71/633 objects degraded (11.216%) 2024-09-17T12:31:32.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:32 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 71/633 objects degraded (11.216%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:32.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:32 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 71/633 objects degraded (11.216%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:32.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:32 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 71/633 objects degraded (11.216%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:33.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:33 smithi016 ceph-mon[87889]: pgmap v149: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 71/633 objects degraded (11.216%) 2024-09-17T12:31:33.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:33 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:33.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:33 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:33.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:33 smithi016 ceph-mon[89784]: pgmap v149: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 71/633 objects degraded (11.216%) 2024-09-17T12:31:33.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:33 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:33.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:33 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:33.526 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:31:33.381+0000 7fb324e49640 -1 osd.5 129 log_to_monitors true 2024-09-17T12:31:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:33 smithi196 ceph-mon[80339]: pgmap v149: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 71/633 objects degraded (11.216%) 2024-09-17T12:31:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:33 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:33.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:33 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:34.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:34.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:34.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[87889]: from='osd.5 [v2:172.21.15.196:6808/1616068550,v1:172.21.15.196:6809/1616068550]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T12:31:34.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[87889]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T12:31:34.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[87889]: pgmap v150: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 71/633 objects degraded (11.216%) 2024-09-17T12:31:34.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:34.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:31:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[89784]: from='osd.5 [v2:172.21.15.196:6808/1616068550,v1:172.21.15.196:6809/1616068550]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T12:31:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[89784]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T12:31:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[89784]: pgmap v150: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 71/633 objects degraded (11.216%) 2024-09-17T12:31:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:34.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:34 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:31:34.719 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:31:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:31:34.335+0000 7fb31cbf3640 -1 osd.5 129 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:31:34.719 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:34 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:34.719 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:34 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:34.719 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:34 smithi196 ceph-mon[80339]: from='osd.5 [v2:172.21.15.196:6808/1616068550,v1:172.21.15.196:6809/1616068550]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T12:31:34.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:34 smithi196 ceph-mon[80339]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T12:31:34.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:34 smithi196 ceph-mon[80339]: pgmap v150: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 71/633 objects degraded (11.216%) 2024-09-17T12:31:34.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:34 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:34.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:34 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:31:35.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:35 smithi016 ceph-mon[87889]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T12:31:35.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:35 smithi016 ceph-mon[87889]: osdmap e132: 8 total, 7 up, 8 in 2024-09-17T12:31:35.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:35 smithi016 ceph-mon[87889]: from='osd.5 [v2:172.21.15.196:6808/1616068550,v1:172.21.15.196:6809/1616068550]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:31:35.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:35 smithi016 ceph-mon[87889]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:31:35.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:35 smithi016 ceph-mon[89784]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T12:31:35.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:35 smithi016 ceph-mon[89784]: osdmap e132: 8 total, 7 up, 8 in 2024-09-17T12:31:35.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:35 smithi016 ceph-mon[89784]: from='osd.5 [v2:172.21.15.196:6808/1616068550,v1:172.21.15.196:6809/1616068550]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:31:35.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:35 smithi016 ceph-mon[89784]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:31:35.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:35 smithi196 ceph-mon[80339]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T12:31:35.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:35 smithi196 ceph-mon[80339]: osdmap e132: 8 total, 7 up, 8 in 2024-09-17T12:31:35.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:35 smithi196 ceph-mon[80339]: from='osd.5 [v2:172.21.15.196:6808/1616068550,v1:172.21.15.196:6809/1616068550]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:31:35.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:35 smithi196 ceph-mon[80339]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:31:36.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:36 smithi196 ceph-mon[80339]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:31:36.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:36 smithi196 ceph-mon[80339]: osd.5 [v2:172.21.15.196:6808/1616068550,v1:172.21.15.196:6809/1616068550] boot 2024-09-17T12:31:36.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:36 smithi196 ceph-mon[80339]: osdmap e133: 8 total, 8 up, 8 in 2024-09-17T12:31:36.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:36 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T12:31:36.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:36 smithi196 ceph-mon[80339]: pgmap v153: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 71/633 objects degraded (11.216%) 2024-09-17T12:31:36.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:36 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:36.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:36 smithi196 ceph-mon[80339]: osdmap e134: 8 total, 8 up, 8 in 2024-09-17T12:31:36.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:31:36] "GET /metrics HTTP/1.1" 200 38625 "" "Prometheus/2.43.0" 2024-09-17T12:31:36.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[87889]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:31:36.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[87889]: osd.5 [v2:172.21.15.196:6808/1616068550,v1:172.21.15.196:6809/1616068550] boot 2024-09-17T12:31:36.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[87889]: osdmap e133: 8 total, 8 up, 8 in 2024-09-17T12:31:36.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T12:31:36.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[87889]: pgmap v153: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 71/633 objects degraded (11.216%) 2024-09-17T12:31:36.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:36.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[87889]: osdmap e134: 8 total, 8 up, 8 in 2024-09-17T12:31:36.673 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[89784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:31:36.673 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[89784]: osd.5 [v2:172.21.15.196:6808/1616068550,v1:172.21.15.196:6809/1616068550] boot 2024-09-17T12:31:36.673 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[89784]: osdmap e133: 8 total, 8 up, 8 in 2024-09-17T12:31:36.673 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T12:31:36.673 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[89784]: pgmap v153: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 71/633 objects degraded (11.216%) 2024-09-17T12:31:36.673 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:36.673 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-mon[89784]: osdmap e134: 8 total, 8 up, 8 in 2024-09-17T12:31:36.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:31:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:31:36.670+0000 7f9d37c73640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-17T12:31:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:37 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:37 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:37 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:31:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:37 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:31:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:37 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:37 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:31:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:37 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:37 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:37.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:37 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:37.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:37 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-17T12:31:37.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:37 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T12:31:37.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:37 smithi196 ceph-mon[80339]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-17T12:31:37.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:37.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:37.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:31:37.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:31:37.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:37.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:31:37.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:37.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:37.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:37.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-17T12:31:37.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T12:31:37.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[89784]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-17T12:31:37.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:37.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:37.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:31:37.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:31:37.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:37.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:31:37.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:37.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:37.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:37.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-17T12:31:37.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T12:31:37.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:37 smithi016 ceph-mon[87889]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-17T12:31:38.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:38 smithi016 ceph-mon[89784]: pgmap v155: 161 pgs: 5 peering, 32 active+undersized, 12 active+undersized+degraded, 112 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 54/633 objects degraded (8.531%) 2024-09-17T12:31:38.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:38 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 54/633 objects degraded (8.531%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:38.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:38 smithi016 ceph-mon[87889]: pgmap v155: 161 pgs: 5 peering, 32 active+undersized, 12 active+undersized+degraded, 112 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 54/633 objects degraded (8.531%) 2024-09-17T12:31:38.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:38 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 54/633 objects degraded (8.531%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:39.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:38 smithi196 ceph-mon[80339]: pgmap v155: 161 pgs: 5 peering, 32 active+undersized, 12 active+undersized+degraded, 112 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 54/633 objects degraded (8.531%) 2024-09-17T12:31:39.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:38 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 54/633 objects degraded (8.531%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T12:31:40.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:40 smithi016 ceph-mon[87889]: pgmap v156: 161 pgs: 5 peering, 7 active+undersized, 1 active+undersized+degraded, 148 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 10/633 objects degraded (1.580%) 2024-09-17T12:31:40.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:40 smithi016 ceph-mon[89784]: pgmap v156: 161 pgs: 5 peering, 7 active+undersized, 1 active+undersized+degraded, 148 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 10/633 objects degraded (1.580%) 2024-09-17T12:31:41.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:40 smithi196 ceph-mon[80339]: pgmap v156: 161 pgs: 5 peering, 7 active+undersized, 1 active+undersized+degraded, 148 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 10/633 objects degraded (1.580%) 2024-09-17T12:31:41.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:41 smithi016 ceph-mon[87889]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/633 objects degraded (1.580%), 1 pg degraded) 2024-09-17T12:31:41.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:41 smithi016 ceph-mon[87889]: Cluster is now healthy 2024-09-17T12:31:41.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:41 smithi016 ceph-mon[89784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/633 objects degraded (1.580%), 1 pg degraded) 2024-09-17T12:31:41.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:41 smithi016 ceph-mon[89784]: Cluster is now healthy 2024-09-17T12:31:42.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:41 smithi196 ceph-mon[80339]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/633 objects degraded (1.580%), 1 pg degraded) 2024-09-17T12:31:42.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:41 smithi196 ceph-mon[80339]: Cluster is now healthy 2024-09-17T12:31:42.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:42 smithi016 ceph-mon[87889]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-09-17T12:31:42.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:42 smithi016 ceph-mon[89784]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-09-17T12:31:43.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:42 smithi196 ceph-mon[80339]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-09-17T12:31:44.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:44 smithi016 ceph-mon[87889]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:31:44.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:44 smithi016 ceph-mon[89784]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:31:45.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:44 smithi196 ceph-mon[80339]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:31:46.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:31:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:31:46] "GET /metrics HTTP/1.1" 200 38625 "" "Prometheus/2.43.0" 2024-09-17T12:31:46.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:46 smithi016 ceph-mon[87889]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T12:31:46.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:46 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:46.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:46 smithi016 ceph-mon[89784]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T12:31:46.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:46 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:47.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:46 smithi196 ceph-mon[80339]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T12:31:47.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:46 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:48.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:48 smithi016 ceph-mon[87889]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:31:48.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:48 smithi016 ceph-mon[89784]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:31:49.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:48 smithi196 ceph-mon[80339]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:31:50.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:49 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:50.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:31:50.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:49 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:50.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:49 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:50.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:31:50.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:49 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:50.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:49 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:50.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:31:50.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:49 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:50.300 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:31:50.890 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:31:50.890 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (9m) 85s ago 10m 25.0M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:31:50.890 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (10m) 18s ago 10m 88.6M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:31:50.890 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (4m) 85s ago 9m 60.9M - 3.8 980a0bfcd684 843db1407344 2024-09-17T12:31:50.891 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (4m) 18s ago 14m 455M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:31:50.891 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (5m) 85s ago 16m 519M - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:31:50.891 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (4m) 85s ago 16m 48.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 4a83b787c605 2024-09-17T12:31:50.891 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (3m) 18s ago 15m 38.2M 2048M 19.3.0-4937-g74984932 6c1fcab85602 dc48dfc9e40c 2024-09-17T12:31:50.891 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (4m) 85s ago 14m 43.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 aa5074acd22a 2024-09-17T12:31:50.891 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (10m) 85s ago 10m 15.0M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:31:50.892 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (10m) 18s ago 10m 15.3M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:31:50.892 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (3m) 85s ago 14m 45.3M 2737M 19.3.0-4937-g74984932 6c1fcab85602 d12e6f26f048 2024-09-17T12:31:50.892 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (2m) 85s ago 13m 53.0M 2737M 19.3.0-4937-g74984932 6c1fcab85602 cd2f6e2964dd 2024-09-17T12:31:50.892 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (2m) 85s ago 13m 42.0M 2737M 19.3.0-4937-g74984932 6c1fcab85602 99430f24b824 2024-09-17T12:31:50.892 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (90s) 85s ago 13m 11.5M 2737M 19.3.0-4937-g74984932 6c1fcab85602 1d4309778a0d 2024-09-17T12:31:50.892 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (56s) 18s ago 12m 43.7M 2992M 19.3.0-4937-g74984932 6c1fcab85602 0c739e0cd187 2024-09-17T12:31:50.892 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (22s) 18s ago 12m 12.2M 2992M 19.3.0-4937-g74984932 6c1fcab85602 71f93c731227 2024-09-17T12:31:50.892 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (12m) 18s ago 11m 63.9M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 d8b32d627f1a 2024-09-17T12:31:50.892 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (11m) 18s ago 11m 67.3M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:31:50.893 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (4m) 18s ago 11m 54.1M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:31:50.893 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (9m) 85s ago 9m 80.7M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:31:50.893 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (9m) 18s ago 9m 81.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:31:50.893 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (9m) 85s ago 9m 83.1M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:31:50.893 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (9m) 18s ago 9m 81.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:31:51.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:51 smithi016 ceph-mon[87889]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:51.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:51 smithi016 ceph-mon[89784]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:51.247 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:31:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:31:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:31:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:31:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:31:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:31:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:31:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:31:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2, 2024-09-17T12:31:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 6 2024-09-17T12:31:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:31:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:31:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:31:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:31:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:31:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 6, 2024-09-17T12:31:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 11 2024-09-17T12:31:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:31:51.249 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:31:51.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:51 smithi196 ceph-mon[80339]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:51.550 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:31:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:31:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:31:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:31:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:31:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-09-17T12:31:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:31:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:31:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "11/23 daemons upgraded", 2024-09-17T12:31:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T12:31:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:31:51.552 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:31:51.901 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-09-17T12:31:52.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:52 smithi016 ceph-mon[87889]: from='client.44381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:52.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:52 smithi016 ceph-mon[87889]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:52.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:52 smithi016 ceph-mon[87889]: from='client.44393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:52.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:52 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/671530503' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:52.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:52 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-17T12:31:52.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:52 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/3937644971' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:31:52.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:52 smithi016 ceph-mon[89784]: from='client.44381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:52.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:52 smithi016 ceph-mon[89784]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:52.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:52 smithi016 ceph-mon[89784]: from='client.44393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:52.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:52 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/671530503' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:52.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:52 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-17T12:31:52.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:52 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/3937644971' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:31:52.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:52 smithi196 ceph-mon[80339]: from='client.44381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:52.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:52 smithi196 ceph-mon[80339]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:52.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:52 smithi196 ceph-mon[80339]: from='client.44393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:52.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:52 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/671530503' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:31:52.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:52 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-17T12:31:52.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:52 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/3937644971' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:31:53.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:53 smithi196 ceph-mon[80339]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:53.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:53 smithi196 ceph-mon[80339]: from='client.44405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:53.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:53 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T12:31:53.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:53 smithi196 ceph-mon[80339]: Upgrade: osd.6 is safe to restart 2024-09-17T12:31:53.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:53 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:53.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:53 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-17T12:31:53.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:53 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:31:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[87889]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[87889]: from='client.44405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T12:31:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[87889]: Upgrade: osd.6 is safe to restart 2024-09-17T12:31:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:53.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-17T12:31:53.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:31:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[89784]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[89784]: from='client.44405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:31:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T12:31:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[89784]: Upgrade: osd.6 is safe to restart 2024-09-17T12:31:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:31:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-17T12:31:53.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:53 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:31:54.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:54 smithi196 ceph-mon[80339]: Upgrade: Updating osd.6 2024-09-17T12:31:54.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:54 smithi196 ceph-mon[80339]: Deploying daemon osd.6 on smithi196 2024-09-17T12:31:54.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:54 smithi016 ceph-mon[87889]: Upgrade: Updating osd.6 2024-09-17T12:31:54.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:54 smithi016 ceph-mon[87889]: Deploying daemon osd.6 on smithi196 2024-09-17T12:31:54.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:54 smithi016 ceph-mon[89784]: Upgrade: Updating osd.6 2024-09-17T12:31:54.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:54 smithi016 ceph-mon[89784]: Deploying daemon osd.6 on smithi196 2024-09-17T12:31:55.184 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:55 smithi196 ceph-mon[80339]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:31:55.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:55 smithi016 ceph-mon[87889]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:31:55.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:55 smithi016 ceph-mon[89784]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:31:56.080 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:55 smithi196 systemd[1]: Stopping Ceph osd.6 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:31:56.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[43518]: 2024-09-17T12:31:56.078+0000 7f5b7e1b9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:31:56.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[43518]: 2024-09-17T12:31:56.078+0000 7f5b7e1b9640 -1 osd.6 134 *** Got signal Terminated *** 2024-09-17T12:31:56.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:56 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[43518]: 2024-09-17T12:31:56.078+0000 7f5b7e1b9640 -1 osd.6 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:31:56.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:31:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:31:56] "GET /metrics HTTP/1.1" 200 38620 "" "Prometheus/2.43.0" 2024-09-17T12:31:57.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:57 smithi196 ceph-mon[80339]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:57.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:57 smithi196 ceph-mon[80339]: osd.6 marked itself down and dead 2024-09-17T12:31:57.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:57 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:57 smithi016 ceph-mon[87889]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:57 smithi016 ceph-mon[87889]: osd.6 marked itself down and dead 2024-09-17T12:31:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:57 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:57 smithi016 ceph-mon[89784]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:31:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:57 smithi016 ceph-mon[89784]: osd.6 marked itself down and dead 2024-09-17T12:31:57.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:57 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:31:57.671 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:57 smithi196 podman[93442]: 2024-09-17 12:31:57.360442074 +0000 UTC m=+1.380266839 container stop d8b32d627f1a5a53ae7f349b1a4a7ce5b3cd1e29ac4f31137ec3eabbefdaa0b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=reef-19e2a2b) 2024-09-17T12:31:57.671 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:57 smithi196 podman[93442]: 2024-09-17 12:31:57.360911931 +0000 UTC m=+1.380736706 container died d8b32d627f1a5a53ae7f349b1a4a7ce5b3cd1e29ac4f31137ec3eabbefdaa0b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-17T12:31:58.005 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:57 smithi196 podman[93442]: 2024-09-17 12:31:57.752958329 +0000 UTC m=+1.772783095 container cleanup d8b32d627f1a5a53ae7f349b1a4a7ce5b3cd1e29ac4f31137ec3eabbefdaa0b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-17T12:31:58.005 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:57 smithi196 bash[93442]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6 2024-09-17T12:31:58.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:58 smithi196 ceph-mon[80339]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:31:58.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:58 smithi196 ceph-mon[80339]: osdmap e135: 8 total, 7 up, 8 in 2024-09-17T12:31:58.280 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:58 smithi196 podman[93471]: 2024-09-17 12:31:58.00402704 +0000 UTC m=+0.632593257 container remove d8b32d627f1a5a53ae7f349b1a4a7ce5b3cd1e29ac4f31137ec3eabbefdaa0b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-17T12:31:58.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:58 smithi016 ceph-mon[87889]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:31:58.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:58 smithi016 ceph-mon[87889]: osdmap e135: 8 total, 7 up, 8 in 2024-09-17T12:31:58.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:58 smithi016 ceph-mon[89784]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:31:58.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:58 smithi016 ceph-mon[89784]: osdmap e135: 8 total, 7 up, 8 in 2024-09-17T12:31:58.700 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:58 smithi196 podman[93557]: 2024-09-17 12:31:58.276474959 +0000 UTC m=+0.016322024 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:31:58.700 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:58 smithi196 podman[93557]: 2024-09-17 12:31:58.379823704 +0000 UTC m=+0.119670750 container create fd4a945f9cdb889fadba40f7778a60cb383d7d0241b4b91f5e3b3c0fb507e7f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-deactivate, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-7498493, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-17T12:31:58.700 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:58 smithi196 podman[93557]: 2024-09-17 12:31:58.615556936 +0000 UTC m=+0.355403999 container init fd4a945f9cdb889fadba40f7778a60cb383d7d0241b4b91f5e3b3c0fb507e7f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=main-7498493, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:31:58.700 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:58 smithi196 podman[93557]: 2024-09-17 12:31:58.618922215 +0000 UTC m=+0.358769263 container start fd4a945f9cdb889fadba40f7778a60cb383d7d0241b4b91f5e3b3c0fb507e7f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-7498493, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-17T12:31:58.700 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:58 smithi196 podman[93557]: 2024-09-17 12:31:58.698994222 +0000 UTC m=+0.438841292 container attach fd4a945f9cdb889fadba40f7778a60cb383d7d0241b4b91f5e3b3c0fb507e7f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=main-7498493, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-17T12:31:59.026 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:58 smithi196 podman[93557]: 2024-09-17 12:31:58.808601519 +0000 UTC m=+0.548448572 container died fd4a945f9cdb889fadba40f7778a60cb383d7d0241b4b91f5e3b3c0fb507e7f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-7498493, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T12:31:59.404 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:59 smithi196 ceph-mon[80339]: pgmap v166: 161 pgs: 14 peering, 10 stale+active+clean, 137 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-09-17T12:31:59.404 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:59 smithi196 ceph-mon[80339]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T12:31:59.404 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:31:59 smithi196 ceph-mon[80339]: osdmap e136: 8 total, 7 up, 8 in 2024-09-17T12:31:59.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:59 smithi016 ceph-mon[87889]: pgmap v166: 161 pgs: 14 peering, 10 stale+active+clean, 137 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-09-17T12:31:59.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:59 smithi016 ceph-mon[87889]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T12:31:59.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:31:59 smithi016 ceph-mon[87889]: osdmap e136: 8 total, 7 up, 8 in 2024-09-17T12:31:59.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:59 smithi016 ceph-mon[89784]: pgmap v166: 161 pgs: 14 peering, 10 stale+active+clean, 137 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-09-17T12:31:59.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:59 smithi016 ceph-mon[89784]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T12:31:59.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:31:59 smithi016 ceph-mon[89784]: osdmap e136: 8 total, 7 up, 8 in 2024-09-17T12:31:59.776 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:59 smithi196 podman[93580]: 2024-09-17 12:31:59.403046713 +0000 UTC m=+0.588842779 container remove fd4a945f9cdb889fadba40f7778a60cb383d7d0241b4b91f5e3b3c0fb507e7f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=main-7498493) 2024-09-17T12:31:59.777 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:59 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.6.service: Deactivated successfully. 2024-09-17T12:31:59.777 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:59 smithi196 systemd[1]: Stopped Ceph osd.6 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:31:59.777 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:59 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.6.service: Consumed 7.081s CPU time. 2024-09-17T12:31:59.777 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:59 smithi196 systemd[1]: Starting Ceph osd.6 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:32:00.095 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:59 smithi196 podman[93680]: 2024-09-17 12:31:59.739802547 +0000 UTC m=+0.016313297 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:32:00.095 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:31:59 smithi196 podman[93680]: 2024-09-17 12:31:59.870919276 +0000 UTC m=+0.147430054 container create dc03f84ff9c2e849bfcbe6dfddcdafb3b4454b71e9ed8c05a21d061a8df4ec28 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-7498493, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:32:00.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:00 smithi016 ceph-mon[87889]: Health check failed: Degraded data redundancy: 28/633 objects degraded (4.423%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:00.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:00 smithi016 ceph-mon[89784]: Health check failed: Degraded data redundancy: 28/633 objects degraded (4.423%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:00.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:00 smithi196 ceph-mon[80339]: Health check failed: Degraded data redundancy: 28/633 objects degraded (4.423%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:00.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:00 smithi196 podman[93680]: 2024-09-17 12:32:00.098287321 +0000 UTC m=+0.374798064 container init dc03f84ff9c2e849bfcbe6dfddcdafb3b4454b71e9ed8c05a21d061a8df4ec28 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-7498493, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-17T12:32:00.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:00 smithi196 podman[93680]: 2024-09-17 12:32:00.101684731 +0000 UTC m=+0.378195466 container start dc03f84ff9c2e849bfcbe6dfddcdafb3b4454b71e9ed8c05a21d061a8df4ec28 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-7498493, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-17T12:32:00.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:00 smithi196 podman[93680]: 2024-09-17 12:32:00.181650231 +0000 UTC m=+0.458160975 container attach dc03f84ff9c2e849bfcbe6dfddcdafb3b4454b71e9ed8c05a21d061a8df4ec28 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-7498493, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:32:00.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:00 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate[93695]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:32:00.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:00 smithi196 bash[93680]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:32:00.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:00 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate[93695]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:32:00.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:00 smithi196 bash[93680]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:32:01.003 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:00 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate[93695]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T12:32:01.003 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:00 smithi196 bash[93680]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T12:32:01.003 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:00 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate[93695]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T12:32:01.003 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:00 smithi196 bash[93680]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T12:32:01.003 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate[93695]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T12:32:01.004 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 bash[93680]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T12:32:01.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:01 smithi196 ceph-mon[80339]: pgmap v168: 161 pgs: 11 active+undersized, 14 peering, 7 stale+active+clean, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 28/633 objects degraded (4.423%) 2024-09-17T12:32:01.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate[93695]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T12:32:01.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 bash[93680]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T12:32:01.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate[93695]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-17T12:32:01.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 bash[93680]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-17T12:32:01.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate[93695]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T12:32:01.278 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 bash[93680]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T12:32:01.278 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate[93695]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-17T12:32:01.278 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 bash[93680]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-17T12:32:01.278 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 podman[93680]: 2024-09-17 12:32:01.030626608 +0000 UTC m=+1.307137355 container died dc03f84ff9c2e849bfcbe6dfddcdafb3b4454b71e9ed8c05a21d061a8df4ec28 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-7498493, org.label-schema.build-date=20240731) 2024-09-17T12:32:01.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:01 smithi016 ceph-mon[87889]: pgmap v168: 161 pgs: 11 active+undersized, 14 peering, 7 stale+active+clean, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 28/633 objects degraded (4.423%) 2024-09-17T12:32:01.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:01 smithi016 ceph-mon[89784]: pgmap v168: 161 pgs: 11 active+undersized, 14 peering, 7 stale+active+clean, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 28/633 objects degraded (4.423%) 2024-09-17T12:32:02.026 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 podman[93835]: 2024-09-17 12:32:01.702277791 +0000 UTC m=+0.665904311 container remove dc03f84ff9c2e849bfcbe6dfddcdafb3b4454b71e9ed8c05a21d061a8df4ec28 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-activate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-7498493, ceph=True, org.label-schema.schema-version=1.0) 2024-09-17T12:32:02.027 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 podman[93895]: 2024-09-17 12:32:01.902911581 +0000 UTC m=+0.101057782 container create 26b8884ca832ed59d4dbb7e1754805b9b59dbf8e12f57bc5961f51391c7953f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-7498493, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:32:02.027 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:01 smithi196 podman[93895]: 2024-09-17 12:32:01.817700454 +0000 UTC m=+0.015846655 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:32:02.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:02 smithi196 podman[93895]: 2024-09-17 12:32:02.221944005 +0000 UTC m=+0.420090213 container init 26b8884ca832ed59d4dbb7e1754805b9b59dbf8e12f57bc5961f51391c7953f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6, RELEASE=main-7498493, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:32:02.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:02 smithi196 podman[93895]: 2024-09-17 12:32:02.225115616 +0000 UTC m=+0.423261815 container start 26b8884ca832ed59d4dbb7e1754805b9b59dbf8e12f57bc5961f51391c7953f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-7498493) 2024-09-17T12:32:02.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:02 smithi196 ceph-osd[93915]: -- 172.21.15.196:0/677574463 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ee4e89a40 con 0x562ee5c58000 2024-09-17T12:32:02.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:02 smithi196 bash[93895]: 26b8884ca832ed59d4dbb7e1754805b9b59dbf8e12f57bc5961f51391c7953f6 2024-09-17T12:32:02.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:02 smithi196 systemd[1]: Started Ceph osd.6 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:32:03.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:03 smithi196 ceph-mon[80339]: pgmap v169: 161 pgs: 24 active+undersized, 14 peering, 15 active+undersized+degraded, 108 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 62/633 objects degraded (9.795%) 2024-09-17T12:32:03.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:03 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:03.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:03 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:03.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:03 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:32:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:03 smithi016 ceph-mon[87889]: pgmap v169: 161 pgs: 24 active+undersized, 14 peering, 15 active+undersized+degraded, 108 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 62/633 objects degraded (9.795%) 2024-09-17T12:32:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:03 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:03 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:03 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:32:03.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:03 smithi016 ceph-mon[89784]: pgmap v169: 161 pgs: 24 active+undersized, 14 peering, 15 active+undersized+degraded, 108 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 62/633 objects degraded (9.795%) 2024-09-17T12:32:03.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:03 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:03.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:03 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:03.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:03 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:32:04.276 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:03 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:32:03.871+0000 7f8d5d966640 -1 Falling back to public interface 2024-09-17T12:32:05.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:04 smithi016 ceph-mon[87889]: pgmap v170: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-17T12:32:05.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:04 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:05.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:04 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:32:05.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:04 smithi016 ceph-mon[87889]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T12:32:05.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:04 smithi016 ceph-mon[89784]: pgmap v170: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-17T12:32:05.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:04 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:05.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:04 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:32:05.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:04 smithi016 ceph-mon[89784]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T12:32:05.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:04 smithi196 ceph-mon[80339]: pgmap v170: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-17T12:32:05.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:04 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:05.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:04 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:32:05.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:04 smithi196 ceph-mon[80339]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T12:32:06.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:32:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:32:06] "GET /metrics HTTP/1.1" 200 38698 "" "Prometheus/2.43.0" 2024-09-17T12:32:07.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:06 smithi016 ceph-mon[87889]: pgmap v171: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-17T12:32:07.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:06 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:07.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:06 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:07.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:06 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:07.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:06 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 75/633 objects degraded (11.848%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:07.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:06 smithi016 ceph-mon[89784]: pgmap v171: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-17T12:32:07.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:06 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:07.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:06 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:07.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:06 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:07.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:06 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 75/633 objects degraded (11.848%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:07.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:06 smithi196 ceph-mon[80339]: pgmap v171: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-17T12:32:07.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:06 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:07.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:06 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:07.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:06 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:07.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:06 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 75/633 objects degraded (11.848%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:07.545 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:07 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:32:07.420+0000 7f8d5d966640 -1 osd.6 134 log_to_monitors true 2024-09-17T12:32:08.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:08 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:08.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:08 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:08.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:08 smithi016 ceph-mon[87889]: from='osd.6 [v2:172.21.15.196:6816/1820566007,v1:172.21.15.196:6817/1820566007]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-17T12:32:08.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:08 smithi016 ceph-mon[87889]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-17T12:32:08.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:08 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:08.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:08 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:08.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:08 smithi016 ceph-mon[89784]: from='osd.6 [v2:172.21.15.196:6816/1820566007,v1:172.21.15.196:6817/1820566007]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-17T12:32:08.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:08 smithi016 ceph-mon[89784]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-17T12:32:08.506 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:08 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:08.507 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:08 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:08.507 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:08 smithi196 ceph-mon[80339]: from='osd.6 [v2:172.21.15.196:6816/1820566007,v1:172.21.15.196:6817/1820566007]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-17T12:32:08.507 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:08 smithi196 ceph-mon[80339]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-17T12:32:08.507 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:32:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:32:08.185+0000 7f8d55710640 -1 osd.6 134 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:32:09.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:09 smithi016 ceph-mon[87889]: pgmap v172: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s; 75/633 objects degraded (11.848%) 2024-09-17T12:32:09.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:09 smithi016 ceph-mon[87889]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T12:32:09.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:09 smithi016 ceph-mon[87889]: osdmap e137: 8 total, 7 up, 8 in 2024-09-17T12:32:09.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:09 smithi016 ceph-mon[87889]: from='osd.6 [v2:172.21.15.196:6816/1820566007,v1:172.21.15.196:6817/1820566007]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:32:09.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:09 smithi016 ceph-mon[87889]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:32:09.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:09 smithi016 ceph-mon[89784]: pgmap v172: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s; 75/633 objects degraded (11.848%) 2024-09-17T12:32:09.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:09 smithi016 ceph-mon[89784]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T12:32:09.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:09 smithi016 ceph-mon[89784]: osdmap e137: 8 total, 7 up, 8 in 2024-09-17T12:32:09.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:09 smithi016 ceph-mon[89784]: from='osd.6 [v2:172.21.15.196:6816/1820566007,v1:172.21.15.196:6817/1820566007]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:32:09.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:09 smithi016 ceph-mon[89784]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:32:09.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:09 smithi196 ceph-mon[80339]: pgmap v172: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s; 75/633 objects degraded (11.848%) 2024-09-17T12:32:09.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:09 smithi196 ceph-mon[80339]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T12:32:09.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:09 smithi196 ceph-mon[80339]: osdmap e137: 8 total, 7 up, 8 in 2024-09-17T12:32:09.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:09 smithi196 ceph-mon[80339]: from='osd.6 [v2:172.21.15.196:6816/1820566007,v1:172.21.15.196:6817/1820566007]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:32:09.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:09 smithi196 ceph-mon[80339]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:32:10.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:10 smithi016 ceph-mon[87889]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:32:10.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:10 smithi016 ceph-mon[87889]: osd.6 [v2:172.21.15.196:6816/1820566007,v1:172.21.15.196:6817/1820566007] boot 2024-09-17T12:32:10.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:10 smithi016 ceph-mon[87889]: osdmap e138: 8 total, 8 up, 8 in 2024-09-17T12:32:10.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:10 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T12:32:10.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:10 smithi016 ceph-mon[87889]: osdmap e139: 8 total, 8 up, 8 in 2024-09-17T12:32:10.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:10 smithi016 ceph-mon[89784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:32:10.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:10 smithi016 ceph-mon[89784]: osd.6 [v2:172.21.15.196:6816/1820566007,v1:172.21.15.196:6817/1820566007] boot 2024-09-17T12:32:10.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:10 smithi016 ceph-mon[89784]: osdmap e138: 8 total, 8 up, 8 in 2024-09-17T12:32:10.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:10 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T12:32:10.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:10 smithi016 ceph-mon[89784]: osdmap e139: 8 total, 8 up, 8 in 2024-09-17T12:32:10.450 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:10 smithi196 ceph-mon[80339]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:32:10.450 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:10 smithi196 ceph-mon[80339]: osd.6 [v2:172.21.15.196:6816/1820566007,v1:172.21.15.196:6817/1820566007] boot 2024-09-17T12:32:10.450 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:10 smithi196 ceph-mon[80339]: osdmap e138: 8 total, 8 up, 8 in 2024-09-17T12:32:10.450 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:10 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T12:32:10.450 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:10 smithi196 ceph-mon[80339]: osdmap e139: 8 total, 8 up, 8 in 2024-09-17T12:32:10.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:32:10 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:32:10.586+0000 7f9d37c73640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-17T12:32:11.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[87889]: pgmap v175: 161 pgs: 16 peering, 24 active+undersized, 13 active+undersized+degraded, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 49/633 objects degraded (7.741%) 2024-09-17T12:32:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:32:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:32:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:11.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-17T12:32:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[89784]: pgmap v175: 161 pgs: 16 peering, 24 active+undersized, 13 active+undersized+degraded, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 49/633 objects degraded (7.741%) 2024-09-17T12:32:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:32:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:32:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:11.443 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:11 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-17T12:32:11.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:11 smithi196 ceph-mon[80339]: pgmap v175: 161 pgs: 16 peering, 24 active+undersized, 13 active+undersized+degraded, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 49/633 objects degraded (7.741%) 2024-09-17T12:32:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:11 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:11 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:11 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:11 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:32:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:11 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:11 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:32:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:11 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:11 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:11 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:11 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-17T12:32:12.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:12 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T12:32:12.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:12 smithi016 ceph-mon[87889]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-17T12:32:12.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:12 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 49/633 objects degraded (7.741%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:12.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:12 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T12:32:12.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:12 smithi016 ceph-mon[89784]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-17T12:32:12.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:12 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 49/633 objects degraded (7.741%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:12.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:12 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T12:32:12.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:12 smithi196 ceph-mon[80339]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-17T12:32:12.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:12 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 49/633 objects degraded (7.741%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:13.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:13 smithi196 ceph-mon[80339]: pgmap v177: 161 pgs: 16 peering, 16 active+undersized, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/633 objects degraded (3.949%) 2024-09-17T12:32:13.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:13 smithi016 ceph-mon[87889]: pgmap v177: 161 pgs: 16 peering, 16 active+undersized, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/633 objects degraded (3.949%) 2024-09-17T12:32:13.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:13 smithi016 ceph-mon[89784]: pgmap v177: 161 pgs: 16 peering, 16 active+undersized, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/633 objects degraded (3.949%) 2024-09-17T12:32:14.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:14 smithi196 ceph-mon[80339]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/633 objects degraded (3.949%), 8 pgs degraded) 2024-09-17T12:32:14.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:14 smithi196 ceph-mon[80339]: Cluster is now healthy 2024-09-17T12:32:14.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:14 smithi016 ceph-mon[87889]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/633 objects degraded (3.949%), 8 pgs degraded) 2024-09-17T12:32:14.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:14 smithi016 ceph-mon[87889]: Cluster is now healthy 2024-09-17T12:32:14.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:14 smithi016 ceph-mon[89784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/633 objects degraded (3.949%), 8 pgs degraded) 2024-09-17T12:32:14.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:14 smithi016 ceph-mon[89784]: Cluster is now healthy 2024-09-17T12:32:15.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:15 smithi196 ceph-mon[80339]: pgmap v178: 161 pgs: 16 peering, 145 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T12:32:15.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:15 smithi016 ceph-mon[87889]: pgmap v178: 161 pgs: 16 peering, 145 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T12:32:15.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:15 smithi016 ceph-mon[89784]: pgmap v178: 161 pgs: 16 peering, 145 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T12:32:16.592 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:32:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:32:16] "GET /metrics HTTP/1.1" 200 38698 "" "Prometheus/2.43.0" 2024-09-17T12:32:17.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:17 smithi196 ceph-mon[80339]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 554 B/s rd, 0 op/s 2024-09-17T12:32:17.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:17 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:17.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:17 smithi016 ceph-mon[87889]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 554 B/s rd, 0 op/s 2024-09-17T12:32:17.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:17 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:17.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:17 smithi016 ceph-mon[89784]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 554 B/s rd, 0 op/s 2024-09-17T12:32:17.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:17 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:19 smithi196 ceph-mon[80339]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:32:19.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:19 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:19.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:19 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:32:19.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:19 smithi016 ceph-mon[87889]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:32:19.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:19 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:19.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:19 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:32:19.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:19 smithi016 ceph-mon[89784]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:32:19.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:19 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:19.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:19 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:32:21.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:21 smithi196 ceph-mon[80339]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T12:32:21.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:21 smithi016 ceph-mon[87889]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T12:32:21.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:21 smithi016 ceph-mon[89784]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T12:32:22.223 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:32:22.811 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:32:22.811 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (10m) 117s ago 11m 25.0M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:32:22.811 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (11m) 16s ago 11m 88.6M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:32:22.811 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (5m) 117s ago 9m 60.9M - 3.8 980a0bfcd684 843db1407344 2024-09-17T12:32:22.811 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (5m) 16s ago 15m 455M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:32:22.811 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (5m) 117s ago 16m 519M - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:32:22.811 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (4m) 117s ago 16m 48.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 4a83b787c605 2024-09-17T12:32:22.811 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (4m) 16s ago 15m 39.5M 2048M 19.3.0-4937-g74984932 6c1fcab85602 dc48dfc9e40c 2024-09-17T12:32:22.811 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (4m) 117s ago 15m 43.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 aa5074acd22a 2024-09-17T12:32:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (10m) 117s ago 10m 15.0M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:32:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (10m) 16s ago 10m 15.5M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:32:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (3m) 117s ago 14m 45.3M 2737M 19.3.0-4937-g74984932 6c1fcab85602 d12e6f26f048 2024-09-17T12:32:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (3m) 117s ago 14m 53.0M 2737M 19.3.0-4937-g74984932 6c1fcab85602 cd2f6e2964dd 2024-09-17T12:32:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (2m) 117s ago 14m 42.0M 2737M 19.3.0-4937-g74984932 6c1fcab85602 99430f24b824 2024-09-17T12:32:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (2m) 117s ago 13m 11.5M 2737M 19.3.0-4937-g74984932 6c1fcab85602 1d4309778a0d 2024-09-17T12:32:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (88s) 16s ago 13m 44.2M 2992M 19.3.0-4937-g74984932 6c1fcab85602 0c739e0cd187 2024-09-17T12:32:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (54s) 16s ago 12m 43.7M 2992M 19.3.0-4937-g74984932 6c1fcab85602 71f93c731227 2024-09-17T12:32:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (20s) 16s ago 12m 11.7M 2992M 19.3.0-4937-g74984932 6c1fcab85602 26b8884ca832 2024-09-17T12:32:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (12m) 16s ago 12m 68.6M 2992M 18.2.4-855-g19e2a2be 980a0bfcd684 675f2e549199 2024-09-17T12:32:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (5m) 16s ago 11m 54.8M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:32:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (9m) 117s ago 9m 80.7M - 18.2.4-855-g19e2a2be 980a0bfcd684 12f859cfc7d4 2024-09-17T12:32:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (9m) 16s ago 9m 82.0M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:32:22.813 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (9m) 117s ago 9m 83.1M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:32:22.813 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (9m) 16s ago 9m 81.6M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:32:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:32:23.163 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:32:23.163 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:32:23.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:32:23.163 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:32:23.163 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:32:23.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:32:23.163 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:32:23.164 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1, 2024-09-17T12:32:23.164 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 7 2024-09-17T12:32:23.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:32:23.164 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:32:23.164 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T12:32:23.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:32:23.164 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:32:23.164 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 5, 2024-09-17T12:32:23.164 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 12 2024-09-17T12:32:23.164 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:32:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:32:23.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:23 smithi016 ceph-mon[87889]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-09-17T12:32:23.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:23 smithi016 ceph-mon[87889]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:23.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:23 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/1213538148' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:23.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:23 smithi016 ceph-mon[89784]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-09-17T12:32:23.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:23 smithi016 ceph-mon[89784]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:23.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:23 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/1213538148' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:23.461 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:32:23.461 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:32:23.461 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:32:23.462 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:32:23.462 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:32:23.462 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-09-17T12:32:23.462 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:32:23.462 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:32:23.462 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "12/23 daemons upgraded", 2024-09-17T12:32:23.462 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T12:32:23.462 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:32:23.462 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:32:23.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:23 smithi196 ceph-mon[80339]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-09-17T12:32:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:23 smithi196 ceph-mon[80339]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:23 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/1213538148' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:23.812 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-09-17T12:32:24.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:24 smithi196 ceph-mon[80339]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:24.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:24 smithi196 ceph-mon[80339]: from='client.44435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:24.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:24 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/2048624247' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:32:24.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:24 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:24.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:24 smithi016 ceph-mon[87889]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:24.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:24 smithi016 ceph-mon[87889]: from='client.44435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:24.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:24 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/2048624247' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:32:24.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:24 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:24.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:24 smithi016 ceph-mon[89784]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:24.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:24 smithi016 ceph-mon[89784]: from='client.44435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:24.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:24 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/2048624247' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:32:24.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:24 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:25.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:25 smithi196 ceph-mon[80339]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:25.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:25 smithi196 ceph-mon[80339]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:32:25.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:25 smithi016 ceph-mon[87889]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:25.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:25 smithi016 ceph-mon[87889]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:32:25.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:25 smithi016 ceph-mon[89784]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:25.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:25 smithi016 ceph-mon[89784]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:32:26.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:26 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-17T12:32:26.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:26 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-17T12:32:26.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:32:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:32:26] "GET /metrics HTTP/1.1" 200 38693 "" "Prometheus/2.43.0" 2024-09-17T12:32:26.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:26 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-17T12:32:27.525 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:27 smithi196 ceph-mon[80339]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T12:32:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:27 smithi196 ceph-mon[80339]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T12:32:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:27 smithi196 ceph-mon[80339]: Upgrade: osd.7 is safe to restart 2024-09-17T12:32:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:27 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:27 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:27 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-17T12:32:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:27 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:27.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[89784]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T12:32:27.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[89784]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T12:32:27.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[89784]: Upgrade: osd.7 is safe to restart 2024-09-17T12:32:27.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:27.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:27.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-17T12:32:27.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:27.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[87889]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T12:32:27.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[87889]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T12:32:27.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[87889]: Upgrade: osd.7 is safe to restart 2024-09-17T12:32:27.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:27.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:27.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-17T12:32:27.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:27 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:28.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:28 smithi196 ceph-mon[80339]: Upgrade: Updating osd.7 2024-09-17T12:32:28.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:28 smithi196 ceph-mon[80339]: Deploying daemon osd.7 on smithi196 2024-09-17T12:32:28.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:28 smithi016 ceph-mon[87889]: Upgrade: Updating osd.7 2024-09-17T12:32:28.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:28 smithi016 ceph-mon[87889]: Deploying daemon osd.7 on smithi196 2024-09-17T12:32:28.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:28 smithi016 ceph-mon[89784]: Upgrade: Updating osd.7 2024-09-17T12:32:28.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:28 smithi016 ceph-mon[89784]: Deploying daemon osd.7 on smithi196 2024-09-17T12:32:29.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:29 smithi196 ceph-mon[80339]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-17T12:32:29.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:29 smithi016 ceph-mon[87889]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-17T12:32:29.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:29 smithi016 ceph-mon[89784]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-17T12:32:30.055 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:29 smithi196 systemd[1]: Stopping Ceph osd.7 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:32:30.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:30 smithi196 ceph-mon[80339]: osd.7 marked itself down and dead 2024-09-17T12:32:30.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:30 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[49245]: 2024-09-17T12:32:30.117+0000 7f205656a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:32:30.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:30 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[49245]: 2024-09-17T12:32:30.117+0000 7f205656a640 -1 osd.7 139 *** Got signal Terminated *** 2024-09-17T12:32:30.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:30 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[49245]: 2024-09-17T12:32:30.117+0000 7f205656a640 -1 osd.7 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:32:30.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:30 smithi016 ceph-mon[87889]: osd.7 marked itself down and dead 2024-09-17T12:32:30.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:30 smithi016 ceph-mon[89784]: osd.7 marked itself down and dead 2024-09-17T12:32:31.002 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:30 smithi196 podman[98198]: 2024-09-17 12:32:30.728972511 +0000 UTC m=+0.748025035 container died 675f2e54919971dc37b676f06b72c1e9c7bc43973ab38f7b5990716b5e102467 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731) 2024-09-17T12:32:31.276 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:31 smithi196 podman[98198]: 2024-09-17 12:32:31.119063705 +0000 UTC m=+1.138116220 container cleanup 675f2e54919971dc37b676f06b72c1e9c7bc43973ab38f7b5990716b5e102467 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-09-17T12:32:31.276 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:31 smithi196 bash[98198]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7 2024-09-17T12:32:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:31 smithi016 ceph-mon[87889]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-17T12:32:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:31 smithi016 ceph-mon[87889]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:32:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:31 smithi016 ceph-mon[87889]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-17T12:32:31.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:31 smithi016 ceph-mon[87889]: osdmap e140: 8 total, 7 up, 8 in 2024-09-17T12:32:31.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:31 smithi016 ceph-mon[89784]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-17T12:32:31.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:31 smithi016 ceph-mon[89784]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:32:31.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:31 smithi016 ceph-mon[89784]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-17T12:32:31.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:31 smithi016 ceph-mon[89784]: osdmap e140: 8 total, 7 up, 8 in 2024-09-17T12:32:31.741 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:31 smithi196 podman[98223]: 2024-09-17 12:32:31.370648664 +0000 UTC m=+0.638577322 container remove 675f2e54919971dc37b676f06b72c1e9c7bc43973ab38f7b5990716b5e102467 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:32:31.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:31 smithi196 ceph-mon[80339]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-17T12:32:31.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:31 smithi196 ceph-mon[80339]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T12:32:31.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:31 smithi196 ceph-mon[80339]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-17T12:32:31.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:31 smithi196 ceph-mon[80339]: osdmap e140: 8 total, 7 up, 8 in 2024-09-17T12:32:32.025 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:31 smithi196 podman[98314]: 2024-09-17 12:32:31.642432601 +0000 UTC m=+0.016217035 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:32:32.025 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:31 smithi196 podman[98314]: 2024-09-17 12:32:31.762988443 +0000 UTC m=+0.136772868 container create c02018a17ade41bd62a6769035068f7784ba8e3f2ad801c54ac68b67da67b8ca (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=main-7498493, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T12:32:32.025 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:32 smithi196 podman[98314]: 2024-09-17 12:32:32.024169885 +0000 UTC m=+0.397954311 container init c02018a17ade41bd62a6769035068f7784ba8e3f2ad801c54ac68b67da67b8ca (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-7498493, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:32:32.277 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:32 smithi196 podman[98314]: 2024-09-17 12:32:32.028482506 +0000 UTC m=+0.402266936 container start c02018a17ade41bd62a6769035068f7784ba8e3f2ad801c54ac68b67da67b8ca (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , RELEASE=main-7498493, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True) 2024-09-17T12:32:32.277 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:32 smithi196 podman[98314]: 2024-09-17 12:32:32.141094579 +0000 UTC m=+0.514879017 container attach c02018a17ade41bd62a6769035068f7784ba8e3f2ad801c54ac68b67da67b8ca (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=main-7498493, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-09-17T12:32:32.277 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:32 smithi196 podman[98314]: 2024-09-17 12:32:32.221521994 +0000 UTC m=+0.595306424 container died c02018a17ade41bd62a6769035068f7784ba8e3f2ad801c54ac68b67da67b8ca (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-7498493, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-17T12:32:32.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:32 smithi016 ceph-mon[87889]: osdmap e141: 8 total, 7 up, 8 in 2024-09-17T12:32:32.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:32 smithi016 ceph-mon[89784]: osdmap e141: 8 total, 7 up, 8 in 2024-09-17T12:32:32.766 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:32 smithi196 ceph-mon[80339]: osdmap e141: 8 total, 7 up, 8 in 2024-09-17T12:32:33.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:32 smithi196 podman[98337]: 2024-09-17 12:32:32.86972517 +0000 UTC m=+0.638932357 container remove c02018a17ade41bd62a6769035068f7784ba8e3f2ad801c54ac68b67da67b8ca (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-7498493, io.buildah.version=1.37.2) 2024-09-17T12:32:33.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:32 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.7.service: Deactivated successfully. 2024-09-17T12:32:33.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:32 smithi196 systemd[1]: Stopped Ceph osd.7 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:32:33.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:32 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.7.service: Consumed 8.497s CPU time. 2024-09-17T12:32:33.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:33 smithi196 ceph-mon[80339]: pgmap v189: 161 pgs: 11 peering, 16 stale+active+clean, 134 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-17T12:32:33.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:33 smithi196 ceph-mon[80339]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-17T12:32:33.448 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:33 smithi196 systemd[1]: Starting Ceph osd.7 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:32:33.448 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:33 smithi196 podman[98438]: 2024-09-17 12:32:33.202988848 +0000 UTC m=+0.015750728 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:32:33.448 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:33 smithi196 podman[98438]: 2024-09-17 12:32:33.446814537 +0000 UTC m=+0.259576404 container create b0aa4d02de7bc3c27b7f3c10af547c1d017c73f0d0f9cbac74d60beb41a528fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=main-7498493) 2024-09-17T12:32:33.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:33 smithi016 ceph-mon[87889]: pgmap v189: 161 pgs: 11 peering, 16 stale+active+clean, 134 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-17T12:32:33.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:33 smithi016 ceph-mon[87889]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-17T12:32:33.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:33 smithi016 ceph-mon[89784]: pgmap v189: 161 pgs: 11 peering, 16 stale+active+clean, 134 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-17T12:32:33.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:33 smithi016 ceph-mon[89784]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-17T12:32:33.758 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:33 smithi196 podman[98438]: 2024-09-17 12:32:33.665729686 +0000 UTC m=+0.478491564 container init b0aa4d02de7bc3c27b7f3c10af547c1d017c73f0d0f9cbac74d60beb41a528fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-7498493, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:32:33.758 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:33 smithi196 podman[98438]: 2024-09-17 12:32:33.668965953 +0000 UTC m=+0.481727821 container start b0aa4d02de7bc3c27b7f3c10af547c1d017c73f0d0f9cbac74d60beb41a528fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-7498493, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:32:33.759 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:33 smithi196 podman[98438]: 2024-09-17 12:32:33.757524291 +0000 UTC m=+0.570286187 container attach b0aa4d02de7bc3c27b7f3c10af547c1d017c73f0d0f9cbac74d60beb41a528fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=main-7498493, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:32:34.026 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate[98453]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:32:34.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:33 smithi196 bash[98438]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:32:34.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate[98453]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:32:34.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:33 smithi196 bash[98438]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T12:32:34.568 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate[98453]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T12:32:34.568 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 bash[98438]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T12:32:34.568 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate[98453]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T12:32:34.568 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 bash[98438]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T12:32:34.569 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate[98453]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T12:32:34.569 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 bash[98438]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T12:32:34.921 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate[98453]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T12:32:34.921 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 bash[98438]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T12:32:34.921 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate[98453]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-17T12:32:34.921 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 bash[98438]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-17T12:32:34.922 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate[98453]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T12:32:34.922 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 bash[98438]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T12:32:34.922 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate[98453]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-17T12:32:34.922 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 bash[98438]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-17T12:32:34.922 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:34 smithi196 podman[98438]: 2024-09-17 12:32:34.595036133 +0000 UTC m=+1.407798011 container died b0aa4d02de7bc3c27b7f3c10af547c1d017c73f0d0f9cbac74d60beb41a528fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-7498493, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-17T12:32:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:34 smithi016 ceph-mon[87889]: pgmap v190: 161 pgs: 29 active+undersized, 11 peering, 4 stale+active+clean, 14 active+undersized+degraded, 103 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 0 B/s wr, 200 op/s; 33/633 objects degraded (5.213%) 2024-09-17T12:32:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:34 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:32:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:34 smithi016 ceph-mon[87889]: Health check failed: Degraded data redundancy: 33/633 objects degraded (5.213%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:34 smithi016 ceph-mon[89784]: pgmap v190: 161 pgs: 29 active+undersized, 11 peering, 4 stale+active+clean, 14 active+undersized+degraded, 103 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 0 B/s wr, 200 op/s; 33/633 objects degraded (5.213%) 2024-09-17T12:32:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:34 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:34 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:32:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:34 smithi016 ceph-mon[89784]: Health check failed: Degraded data redundancy: 33/633 objects degraded (5.213%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:35.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:34 smithi196 ceph-mon[80339]: pgmap v190: 161 pgs: 29 active+undersized, 11 peering, 4 stale+active+clean, 14 active+undersized+degraded, 103 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 0 B/s wr, 200 op/s; 33/633 objects degraded (5.213%) 2024-09-17T12:32:35.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:34 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:35.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:34 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:32:35.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:34 smithi196 ceph-mon[80339]: Health check failed: Degraded data redundancy: 33/633 objects degraded (5.213%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:35.526 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:35 smithi196 podman[98579]: 2024-09-17 12:32:35.222700458 +0000 UTC m=+0.621647594 container remove b0aa4d02de7bc3c27b7f3c10af547c1d017c73f0d0f9cbac74d60beb41a528fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-activate, RELEASE=main-7498493, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-17T12:32:35.526 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:35 smithi196 podman[98638]: 2024-09-17 12:32:35.331425003 +0000 UTC m=+0.016034618 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:32:35.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:35 smithi196 podman[98638]: 2024-09-17 12:32:35.440144687 +0000 UTC m=+0.124754336 container create 3ebdff7d6386765cc35732e772ff4c15f1d9a036b883ae35a103bcbf72d0f9a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-7498493, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0) 2024-09-17T12:32:36.026 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:35 smithi196 podman[98638]: 2024-09-17 12:32:35.667698788 +0000 UTC m=+0.352308406 container init 3ebdff7d6386765cc35732e772ff4c15f1d9a036b883ae35a103bcbf72d0f9a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=main-7498493, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-17T12:32:36.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:35 smithi196 podman[98638]: 2024-09-17 12:32:35.670791136 +0000 UTC m=+0.355400744 container start 3ebdff7d6386765cc35732e772ff4c15f1d9a036b883ae35a103bcbf72d0f9a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-7498493, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:32:36.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:35 smithi196 ceph-osd[98656]: -- 172.21.15.196:0/1738219412 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563257237a40 con 0x563258006000 2024-09-17T12:32:36.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:35 smithi196 bash[98638]: 3ebdff7d6386765cc35732e772ff4c15f1d9a036b883ae35a103bcbf72d0f9a8 2024-09-17T12:32:36.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:35 smithi196 systemd[1]: Started Ceph osd.7 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:32:36.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:32:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:32:36] "GET /metrics HTTP/1.1" 200 38766 "" "Prometheus/2.43.0" 2024-09-17T12:32:37.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:36 smithi196 ceph-mon[80339]: pgmap v191: 161 pgs: 36 active+undersized, 11 peering, 22 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 0 B/s wr, 215 op/s; 53/633 objects degraded (8.373%) 2024-09-17T12:32:37.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:36 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:37.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:36 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:37.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:36 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:32:37.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:36 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:37.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:36 smithi016 ceph-mon[87889]: pgmap v191: 161 pgs: 36 active+undersized, 11 peering, 22 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 0 B/s wr, 215 op/s; 53/633 objects degraded (8.373%) 2024-09-17T12:32:37.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:36 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:37.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:36 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:37.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:36 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:32:37.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:36 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:36 smithi016 ceph-mon[89784]: pgmap v191: 161 pgs: 36 active+undersized, 11 peering, 22 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 0 B/s wr, 215 op/s; 53/633 objects degraded (8.373%) 2024-09-17T12:32:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:36 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:36 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:36 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:32:37.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:36 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:37.321 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:37 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:32:37.318+0000 7fa2d1a2a640 -1 Falling back to public interface 2024-09-17T12:32:38.152 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:37 smithi196 ceph-mon[80339]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-17T12:32:38.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:37 smithi016 ceph-mon[87889]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-17T12:32:38.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:37 smithi016 ceph-mon[89784]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-17T12:32:39.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:38 smithi016 ceph-mon[87889]: pgmap v192: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 0 B/s wr, 223 op/s; 86/633 objects degraded (13.586%) 2024-09-17T12:32:39.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:38 smithi016 ceph-mon[89784]: pgmap v192: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 0 B/s wr, 223 op/s; 86/633 objects degraded (13.586%) 2024-09-17T12:32:39.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:38 smithi196 ceph-mon[80339]: pgmap v192: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 0 B/s wr, 223 op/s; 86/633 objects degraded (13.586%) 2024-09-17T12:32:41.226 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:40 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:32:40.890+0000 7fa2d1a2a640 -1 osd.7 139 log_to_monitors true 2024-09-17T12:32:41.226 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:41 smithi196 ceph-mon[80339]: pgmap v193: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 0 B/s wr, 191 op/s; 86/633 objects degraded (13.586%) 2024-09-17T12:32:41.226 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:41 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:41.226 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:41 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:41.226 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:41 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:41.227 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:41 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:41.227 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:41 smithi196 ceph-mon[80339]: from='osd.7 [v2:172.21.15.196:6824/631076702,v1:172.21.15.196:6825/631076702]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T12:32:41.227 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:41 smithi196 ceph-mon[80339]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T12:32:41.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[87889]: pgmap v193: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 0 B/s wr, 191 op/s; 86/633 objects degraded (13.586%) 2024-09-17T12:32:41.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:41.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:41.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:41.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:41.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[87889]: from='osd.7 [v2:172.21.15.196:6824/631076702,v1:172.21.15.196:6825/631076702]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T12:32:41.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[87889]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T12:32:41.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[89784]: pgmap v193: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 0 B/s wr, 191 op/s; 86/633 objects degraded (13.586%) 2024-09-17T12:32:41.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:41.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:41.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:41.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:41.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[89784]: from='osd.7 [v2:172.21.15.196:6824/631076702,v1:172.21.15.196:6825/631076702]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T12:32:41.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:41 smithi016 ceph-mon[89784]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T12:32:42.026 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:32:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:32:41.618+0000 7fa2c97d4640 -1 osd.7 139 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T12:32:42.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:42 smithi016 ceph-mon[87889]: pgmap v194: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 165 op/s; 86/633 objects degraded (13.586%) 2024-09-17T12:32:42.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:42 smithi016 ceph-mon[87889]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T12:32:42.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:42 smithi016 ceph-mon[87889]: from='osd.7 [v2:172.21.15.196:6824/631076702,v1:172.21.15.196:6825/631076702]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:32:42.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:42 smithi016 ceph-mon[87889]: osdmap e142: 8 total, 7 up, 8 in 2024-09-17T12:32:42.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:42 smithi016 ceph-mon[87889]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:32:42.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:42 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 86/633 objects degraded (13.586%), 29 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:42.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:42 smithi016 ceph-mon[89784]: pgmap v194: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 165 op/s; 86/633 objects degraded (13.586%) 2024-09-17T12:32:42.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:42 smithi016 ceph-mon[89784]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T12:32:42.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:42 smithi016 ceph-mon[89784]: from='osd.7 [v2:172.21.15.196:6824/631076702,v1:172.21.15.196:6825/631076702]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:32:42.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:42 smithi016 ceph-mon[89784]: osdmap e142: 8 total, 7 up, 8 in 2024-09-17T12:32:42.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:42 smithi016 ceph-mon[89784]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:32:42.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:42 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 86/633 objects degraded (13.586%), 29 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:43.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:42 smithi196 ceph-mon[80339]: pgmap v194: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 165 op/s; 86/633 objects degraded (13.586%) 2024-09-17T12:32:43.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:42 smithi196 ceph-mon[80339]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T12:32:43.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:42 smithi196 ceph-mon[80339]: from='osd.7 [v2:172.21.15.196:6824/631076702,v1:172.21.15.196:6825/631076702]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:32:43.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:42 smithi196 ceph-mon[80339]: osdmap e142: 8 total, 7 up, 8 in 2024-09-17T12:32:43.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:42 smithi196 ceph-mon[80339]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-17T12:32:43.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:42 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 86/633 objects degraded (13.586%), 29 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:43.722 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:43 smithi196 ceph-mon[80339]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:32:43.722 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:43 smithi196 ceph-mon[80339]: osd.7 [v2:172.21.15.196:6824/631076702,v1:172.21.15.196:6825/631076702] boot 2024-09-17T12:32:43.722 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:43 smithi196 ceph-mon[80339]: osdmap e143: 8 total, 8 up, 8 in 2024-09-17T12:32:43.722 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:43 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T12:32:43.722 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:43 smithi196 ceph-mon[80339]: osdmap e144: 8 total, 8 up, 8 in 2024-09-17T12:32:43.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:43 smithi016 ceph-mon[87889]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:32:43.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:43 smithi016 ceph-mon[87889]: osd.7 [v2:172.21.15.196:6824/631076702,v1:172.21.15.196:6825/631076702] boot 2024-09-17T12:32:43.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:43 smithi016 ceph-mon[87889]: osdmap e143: 8 total, 8 up, 8 in 2024-09-17T12:32:43.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:43 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T12:32:43.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:43 smithi016 ceph-mon[87889]: osdmap e144: 8 total, 8 up, 8 in 2024-09-17T12:32:43.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:43 smithi016 ceph-mon[89784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T12:32:43.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:43 smithi016 ceph-mon[89784]: osd.7 [v2:172.21.15.196:6824/631076702,v1:172.21.15.196:6825/631076702] boot 2024-09-17T12:32:43.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:43 smithi016 ceph-mon[89784]: osdmap e143: 8 total, 8 up, 8 in 2024-09-17T12:32:43.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:43 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T12:32:43.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:43 smithi016 ceph-mon[89784]: osdmap e144: 8 total, 8 up, 8 in 2024-09-17T12:32:45.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: pgmap v197: 161 pgs: 10 peering, 33 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s; 83/633 objects degraded (13.112%) 2024-09-17T12:32:45.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:45.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:45.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:45.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:32:45.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:45.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:32:45.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: Upgrade: Setting container_image for all osd 2024-09-17T12:32:45.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:45.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.0"} : dispatch 2024-09-17T12:32:45.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.0"} : dispatch 2024-09-17T12:32:45.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-17T12:32:45.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.1"} : dispatch 2024-09-17T12:32:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.1"} : dispatch 2024-09-17T12:32:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-17T12:32:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.2"} : dispatch 2024-09-17T12:32:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.2"} : dispatch 2024-09-17T12:32:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-17T12:32:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.3"} : dispatch 2024-09-17T12:32:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.3"} : dispatch 2024-09-17T12:32:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-17T12:32:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.4"} : dispatch 2024-09-17T12:32:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.4"} : dispatch 2024-09-17T12:32:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-17T12:32:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.5"} : dispatch 2024-09-17T12:32:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.5"} : dispatch 2024-09-17T12:32:45.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-17T12:32:45.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.6"} : dispatch 2024-09-17T12:32:45.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.6"} : dispatch 2024-09-17T12:32:45.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-17T12:32:45.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.7"} : dispatch 2024-09-17T12:32:45.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.7"} : dispatch 2024-09-17T12:32:45.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-17T12:32:45.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: Upgrade: Setting require_osd_release to 19 squid 2024-09-17T12:32:45.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd require-osd-release", "release": "squid"} : dispatch 2024-09-17T12:32:45.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:44 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "osd require-osd-release", "release": "squid"} : dispatch 2024-09-17T12:32:45.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: pgmap v197: 161 pgs: 10 peering, 33 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s; 83/633 objects degraded (13.112%) 2024-09-17T12:32:45.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:45.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:45.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:45.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:32:45.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:45.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:32:45.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: Upgrade: Setting container_image for all osd 2024-09-17T12:32:45.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:45.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.0"} : dispatch 2024-09-17T12:32:45.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.0"} : dispatch 2024-09-17T12:32:45.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-17T12:32:45.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.1"} : dispatch 2024-09-17T12:32:45.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.1"} : dispatch 2024-09-17T12:32:45.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-17T12:32:45.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.2"} : dispatch 2024-09-17T12:32:45.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.2"} : dispatch 2024-09-17T12:32:45.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-17T12:32:45.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.3"} : dispatch 2024-09-17T12:32:45.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.3"} : dispatch 2024-09-17T12:32:45.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-17T12:32:45.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.4"} : dispatch 2024-09-17T12:32:45.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.4"} : dispatch 2024-09-17T12:32:45.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: pgmap v197: 161 pgs: 10 peering, 33 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s; 83/633 objects degraded (13.112%) 2024-09-17T12:32:45.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:45.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:45.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:45.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:32:45.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:45.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: Upgrade: Setting container_image for all osd 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.0"} : dispatch 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.0"} : dispatch 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.1"} : dispatch 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.1"} : dispatch 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.2"} : dispatch 2024-09-17T12:32:45.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.2"} : dispatch 2024-09-17T12:32:45.196 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-17T12:32:45.196 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.3"} : dispatch 2024-09-17T12:32:45.196 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.3"} : dispatch 2024-09-17T12:32:45.196 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-17T12:32:45.196 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.4"} : dispatch 2024-09-17T12:32:45.196 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.4"} : dispatch 2024-09-17T12:32:45.196 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-17T12:32:45.196 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.5"} : dispatch 2024-09-17T12:32:45.196 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.5"} : dispatch 2024-09-17T12:32:45.196 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-17T12:32:45.196 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.6"} : dispatch 2024-09-17T12:32:45.196 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.6"} : dispatch 2024-09-17T12:32:45.196 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-17T12:32:45.197 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.7"} : dispatch 2024-09-17T12:32:45.197 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.7"} : dispatch 2024-09-17T12:32:45.197 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-17T12:32:45.197 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: Upgrade: Setting require_osd_release to 19 squid 2024-09-17T12:32:45.197 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd require-osd-release", "release": "squid"} : dispatch 2024-09-17T12:32:45.197 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "osd require-osd-release", "release": "squid"} : dispatch 2024-09-17T12:32:45.197 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-17T12:32:45.197 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.5"} : dispatch 2024-09-17T12:32:45.197 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.5"} : dispatch 2024-09-17T12:32:45.197 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-17T12:32:45.197 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.6"} : dispatch 2024-09-17T12:32:45.197 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.6"} : dispatch 2024-09-17T12:32:45.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-17T12:32:45.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.7"} : dispatch 2024-09-17T12:32:45.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.7"} : dispatch 2024-09-17T12:32:45.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-17T12:32:45.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: Upgrade: Setting require_osd_release to 19 squid 2024-09-17T12:32:45.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd require-osd-release", "release": "squid"} : dispatch 2024-09-17T12:32:45.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:44 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "osd require-osd-release", "release": "squid"} : dispatch 2024-09-17T12:32:45.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:45 smithi016 ceph-mon[87889]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-17T12:32:45.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:45 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-17T12:32:45.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:45 smithi016 ceph-mon[87889]: osdmap e145: 8 total, 8 up, 8 in 2024-09-17T12:32:45.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:45 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:45 smithi016 ceph-mon[87889]: Upgrade: Setting container_image for all mds 2024-09-17T12:32:45.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:45 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:45.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:45 smithi016 ceph-mon[89784]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-17T12:32:45.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:45 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-17T12:32:45.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:45 smithi016 ceph-mon[89784]: osdmap e145: 8 total, 8 up, 8 in 2024-09-17T12:32:45.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:45 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:45.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:45 smithi016 ceph-mon[89784]: Upgrade: Setting container_image for all mds 2024-09-17T12:32:45.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:45 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:46.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:45 smithi196 ceph-mon[80339]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-17T12:32:46.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:45 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-17T12:32:46.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:45 smithi196 ceph-mon[80339]: osdmap e145: 8 total, 8 up, 8 in 2024-09-17T12:32:46.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:45 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:46.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:45 smithi196 ceph-mon[80339]: Upgrade: Setting container_image for all mds 2024-09-17T12:32:46.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:45 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:46.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:32:46] "GET /metrics HTTP/1.1" 200 38766 "" "Prometheus/2.43.0" 2024-09-17T12:32:46.991 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[87889]: pgmap v200: 161 pgs: 10 peering, 29 active+undersized, 19 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 71/633 objects degraded (11.216%); 0 B/s, 0 objects/s recovering 2024-09-17T12:32:46.992 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:46.992 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:46.992 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:46.992 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:46.992 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:46.992 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[87889]: Health check update: Degraded data redundancy: 71/633 objects degraded (11.216%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:46.992 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[89784]: pgmap v200: 161 pgs: 10 peering, 29 active+undersized, 19 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 71/633 objects degraded (11.216%); 0 B/s, 0 objects/s recovering 2024-09-17T12:32:46.992 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:46.992 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:46.993 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:46.993 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:46.993 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:46.993 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:46 smithi016 ceph-mon[89784]: Health check update: Degraded data redundancy: 71/633 objects degraded (11.216%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:47.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:46 smithi196 ceph-mon[80339]: pgmap v200: 161 pgs: 10 peering, 29 active+undersized, 19 active+undersized+degraded, 103 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 71/633 objects degraded (11.216%); 0 B/s, 0 objects/s recovering 2024-09-17T12:32:47.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:46 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:47.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:46 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:47.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:46 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:47.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:46 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.tquwud", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:47.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:46 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:47.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:46 smithi196 ceph-mon[80339]: Health check update: Degraded data redundancy: 71/633 objects degraded (11.216%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T12:32:48.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:47 smithi016 ceph-mon[87889]: Upgrade: Updating rgw.foo.smithi016.tquwud (1/4) 2024-09-17T12:32:48.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:47 smithi016 ceph-mon[87889]: Deploying daemon rgw.foo.smithi016.tquwud on smithi016 2024-09-17T12:32:48.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:47 smithi016 ceph-mon[89784]: Upgrade: Updating rgw.foo.smithi016.tquwud (1/4) 2024-09-17T12:32:48.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:47 smithi016 ceph-mon[89784]: Deploying daemon rgw.foo.smithi016.tquwud on smithi016 2024-09-17T12:32:48.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:47 smithi196 ceph-mon[80339]: Upgrade: Updating rgw.foo.smithi016.tquwud (1/4) 2024-09-17T12:32:48.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:47 smithi196 ceph-mon[80339]: Deploying daemon rgw.foo.smithi016.tquwud on smithi016 2024-09-17T12:32:49.134 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:48 smithi016 ceph-mon[87889]: pgmap v201: 161 pgs: 10 peering, 151 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 8 objects/s recovering 2024-09-17T12:32:49.134 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:48 smithi016 ceph-mon[87889]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/633 objects degraded (11.216%), 19 pgs degraded) 2024-09-17T12:32:49.134 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:48 smithi016 ceph-mon[87889]: Cluster is now healthy 2024-09-17T12:32:49.134 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:48 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:49.134 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:48 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:32:49.134 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:48 smithi016 ceph-mon[89784]: pgmap v201: 161 pgs: 10 peering, 151 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 8 objects/s recovering 2024-09-17T12:32:49.134 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:48 smithi016 ceph-mon[89784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/633 objects degraded (11.216%), 19 pgs degraded) 2024-09-17T12:32:49.135 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:48 smithi016 ceph-mon[89784]: Cluster is now healthy 2024-09-17T12:32:49.135 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:48 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:49.135 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:48 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:32:49.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:48 smithi196 ceph-mon[80339]: pgmap v201: 161 pgs: 10 peering, 151 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 8 objects/s recovering 2024-09-17T12:32:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:48 smithi196 ceph-mon[80339]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/633 objects degraded (11.216%), 19 pgs degraded) 2024-09-17T12:32:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:48 smithi196 ceph-mon[80339]: Cluster is now healthy 2024-09-17T12:32:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:48 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:49.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:48 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:32:51.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:51 smithi016 ceph-mon[89784]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 293 B/s rd, 0 op/s; 0 B/s, 7 objects/s recovering 2024-09-17T12:32:51.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:51 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:51.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:51 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:51.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:51 smithi016 ceph-mon[87889]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 293 B/s rd, 0 op/s; 0 B/s, 7 objects/s recovering 2024-09-17T12:32:51.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:51 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:51.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:51 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:51.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:51 smithi196 ceph-mon[80339]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 293 B/s rd, 0 op/s; 0 B/s, 7 objects/s recovering 2024-09-17T12:32:51.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:51 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:51.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:51 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:52.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[87889]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 127 B/s wr, 27 op/s; 0 B/s, 6 objects/s recovering 2024-09-17T12:32:52.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[87889]: Upgrade: Updating rgw.smpl.smithi016.qdkyfn (2/4) 2024-09-17T12:32:52.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:52.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:52.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:52.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:52.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[87889]: Deploying daemon rgw.smpl.smithi016.qdkyfn on smithi016 2024-09-17T12:32:52.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[89784]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 127 B/s wr, 27 op/s; 0 B/s, 6 objects/s recovering 2024-09-17T12:32:52.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[89784]: Upgrade: Updating rgw.smpl.smithi016.qdkyfn (2/4) 2024-09-17T12:32:52.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:52.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:52.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:52.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:52.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:52 smithi016 ceph-mon[89784]: Deploying daemon rgw.smpl.smithi016.qdkyfn on smithi016 2024-09-17T12:32:53.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:52 smithi196 ceph-mon[80339]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 127 B/s wr, 27 op/s; 0 B/s, 6 objects/s recovering 2024-09-17T12:32:53.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:52 smithi196 ceph-mon[80339]: Upgrade: Updating rgw.smpl.smithi016.qdkyfn (2/4) 2024-09-17T12:32:53.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:52 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:53.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:52 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:53.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:52 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.qdkyfn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:53.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:52 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:53.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:52 smithi196 ceph-mon[80339]: Deploying daemon rgw.smpl.smithi016.qdkyfn on smithi016 2024-09-17T12:32:54.132 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:32:54.726 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:32:54.727 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (11m) 2m ago 11m 25.0M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:32:54.727 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (11m) 14s ago 11m 88.6M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:32:54.727 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (5m) 2m ago 10m 60.9M - 3.8 980a0bfcd684 843db1407344 2024-09-17T12:32:54.727 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (5m) 14s ago 15m 455M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:32:54.727 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (6m) 2m ago 17m 519M - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:32:54.727 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (5m) 2m ago 17m 48.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 4a83b787c605 2024-09-17T12:32:54.727 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (4m) 14s ago 16m 39.4M 2048M 19.3.0-4937-g74984932 6c1fcab85602 dc48dfc9e40c 2024-09-17T12:32:54.727 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (5m) 2m ago 15m 43.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 aa5074acd22a 2024-09-17T12:32:54.727 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (11m) 2m ago 11m 15.0M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:32:54.727 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (11m) 14s ago 11m 16.5M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:32:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (4m) 2m ago 15m 45.3M 2737M 19.3.0-4937-g74984932 6c1fcab85602 d12e6f26f048 2024-09-17T12:32:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (3m) 2m ago 15m 53.0M 2737M 19.3.0-4937-g74984932 6c1fcab85602 cd2f6e2964dd 2024-09-17T12:32:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (3m) 2m ago 14m 42.0M 2737M 19.3.0-4937-g74984932 6c1fcab85602 99430f24b824 2024-09-17T12:32:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (2m) 2m ago 14m 11.5M 2737M 19.3.0-4937-g74984932 6c1fcab85602 1d4309778a0d 2024-09-17T12:32:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (2m) 14s ago 13m 49.3M 2992M 19.3.0-4937-g74984932 6c1fcab85602 0c739e0cd187 2024-09-17T12:32:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (86s) 14s ago 13m 46.4M 2992M 19.3.0-4937-g74984932 6c1fcab85602 71f93c731227 2024-09-17T12:32:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (52s) 14s ago 13m 44.5M 2992M 19.3.0-4937-g74984932 6c1fcab85602 26b8884ca832 2024-09-17T12:32:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (19s) 14s ago 12m 12.0M 2992M 19.3.0-4937-g74984932 6c1fcab85602 3ebdff7d6386 2024-09-17T12:32:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (5m) 14s ago 12m 57.2M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:32:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 starting - - - - 2024-09-17T12:32:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (10m) 14s ago 10m 82.5M - 18.2.4-855-g19e2a2be 980a0bfcd684 564702d4d8f8 2024-09-17T12:32:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (10m) 2m ago 10m 83.1M - 18.2.4-855-g19e2a2be 980a0bfcd684 62d0bfb45df0 2024-09-17T12:32:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (10m) 14s ago 10m 81.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 734154b71f2f 2024-09-17T12:32:54.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:54 smithi016 ceph-mon[87889]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 102 B/s wr, 79 op/s; 0 B/s, 5 objects/s recovering 2024-09-17T12:32:54.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:54 smithi016 ceph-mon[87889]: from='client.44486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:54.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:54 smithi016 ceph-mon[89784]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 102 B/s wr, 79 op/s; 0 B/s, 5 objects/s recovering 2024-09-17T12:32:54.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:54 smithi016 ceph-mon[89784]: from='client.44486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:55.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:54 smithi196 ceph-mon[80339]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 102 B/s wr, 79 op/s; 0 B/s, 5 objects/s recovering 2024-09-17T12:32:55.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:54 smithi196 ceph-mon[80339]: from='client.44486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:55.102 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:32:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:32:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:32:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:32:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:32:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:32:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:32:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:32:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 8 2024-09-17T12:32:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:32:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:32:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3, 2024-09-17T12:32:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 1 2024-09-17T12:32:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:32:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:32:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3, 2024-09-17T12:32:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 14 2024-09-17T12:32:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:32:55.103 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:32:55.422 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:32:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:32:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:32:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:32:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:32:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-09-17T12:32:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "osd", 2024-09-17T12:32:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:32:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:32:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "13/23 daemons upgraded", 2024-09-17T12:32:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading rgw daemons", 2024-09-17T12:32:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:32:55.423 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:32:55.864 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-09-17T12:32:55.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:55 smithi016 ceph-mon[87889]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:55.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:55 smithi016 ceph-mon[87889]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:55.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:55 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/3652381066' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:55.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:55 smithi016 ceph-mon[89784]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:55.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:55 smithi016 ceph-mon[89784]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:55.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:55 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/3652381066' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:56.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:55 smithi196 ceph-mon[80339]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:56.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:55 smithi196 ceph-mon[80339]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:56.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:55 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/3652381066' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:32:56.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:32:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:32:56] "GET /metrics HTTP/1.1" 200 38767 "" "Prometheus/2.43.0" 2024-09-17T12:32:57.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:56 smithi196 ceph-mon[80339]: from='client.44510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:57.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:56 smithi196 ceph-mon[80339]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 96 B/s wr, 94 op/s; 0 B/s, 4 objects/s recovering 2024-09-17T12:32:57.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:56 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:57.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:56 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:57.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:56 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/3347920129' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:32:57.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:56 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:57.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:56 smithi016 ceph-mon[89784]: from='client.44510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:57.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:56 smithi016 ceph-mon[89784]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 96 B/s wr, 94 op/s; 0 B/s, 4 objects/s recovering 2024-09-17T12:32:57.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:56 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:57.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:56 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:57.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:56 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/3347920129' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:32:57.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:56 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:56 smithi016 ceph-mon[87889]: from='client.44510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:32:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:56 smithi016 ceph-mon[87889]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 96 B/s wr, 94 op/s; 0 B/s, 4 objects/s recovering 2024-09-17T12:32:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:56 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:56 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:56 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/3347920129' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:32:57.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:56 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:32:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:58 smithi196 ceph-mon[80339]: Upgrade: Updating rgw.foo.smithi196.wvbocw (3/4) 2024-09-17T12:32:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:58 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:58 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:58 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:58 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:58.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:58 smithi016 ceph-mon[87889]: Upgrade: Updating rgw.foo.smithi196.wvbocw (3/4) 2024-09-17T12:32:58.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:58 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:58.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:58 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:58.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:58 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:58.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:58 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:58.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:58 smithi016 ceph-mon[89784]: Upgrade: Updating rgw.foo.smithi196.wvbocw (3/4) 2024-09-17T12:32:58.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:58 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:58.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:58 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:58.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:58 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wvbocw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:32:58.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:58 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:32:59.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:59 smithi016 ceph-mon[87889]: Deploying daemon rgw.foo.smithi196.wvbocw on smithi196 2024-09-17T12:32:59.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:59 smithi016 ceph-mon[87889]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 85 B/s wr, 126 op/s; 0 B/s, 3 objects/s recovering 2024-09-17T12:32:59.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:32:59 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:59.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:59 smithi016 ceph-mon[89784]: Deploying daemon rgw.foo.smithi196.wvbocw on smithi196 2024-09-17T12:32:59.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:59 smithi016 ceph-mon[89784]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 85 B/s wr, 126 op/s; 0 B/s, 3 objects/s recovering 2024-09-17T12:32:59.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:32:59 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:32:59.747 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:59 smithi196 ceph-mon[80339]: Deploying daemon rgw.foo.smithi196.wvbocw on smithi196 2024-09-17T12:32:59.747 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:59 smithi196 ceph-mon[80339]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 85 B/s wr, 126 op/s; 0 B/s, 3 objects/s recovering 2024-09-17T12:32:59.748 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:32:59 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:00.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:00 smithi196 ceph-mon[80339]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 85 B/s wr, 144 op/s 2024-09-17T12:33:00.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:00 smithi016 ceph-mon[87889]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 85 B/s wr, 144 op/s 2024-09-17T12:33:00.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:00 smithi016 ceph-mon[89784]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 85 B/s wr, 144 op/s 2024-09-17T12:33:02.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:02 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:02.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:02 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:02.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:02 smithi016 ceph-mon[87889]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-17T12:33:02.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:02 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:02.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:02 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:02.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:02 smithi016 ceph-mon[89784]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-17T12:33:02.733 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:02 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:02.733 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:02 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:02.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:02 smithi196 ceph-mon[80339]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-17T12:33:04.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:03 smithi196 ceph-mon[80339]: Upgrade: Updating rgw.smpl.smithi196.ghtvuz (4/4) 2024-09-17T12:33:04.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:03 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:04.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:03 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:33:04.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:03 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:33:04.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:03 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:04.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:03 smithi196 ceph-mon[80339]: Deploying daemon rgw.smpl.smithi196.ghtvuz on smithi196 2024-09-17T12:33:04.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:03 smithi016 ceph-mon[89784]: Upgrade: Updating rgw.smpl.smithi196.ghtvuz (4/4) 2024-09-17T12:33:04.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:03 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:04.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:03 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:33:04.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:03 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:33:04.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:03 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:04.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:03 smithi016 ceph-mon[89784]: Deploying daemon rgw.smpl.smithi196.ghtvuz on smithi196 2024-09-17T12:33:04.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:03 smithi016 ceph-mon[87889]: Upgrade: Updating rgw.smpl.smithi196.ghtvuz (4/4) 2024-09-17T12:33:04.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:03 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:04.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:03 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:33:04.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:03 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.ghtvuz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-17T12:33:04.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:03 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:04.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:03 smithi016 ceph-mon[87889]: Deploying daemon rgw.smpl.smithi196.ghtvuz on smithi196 2024-09-17T12:33:04.903 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:04 smithi196 ceph-mon[80339]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 85 B/s wr, 195 op/s 2024-09-17T12:33:04.903 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:04 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:33:05.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:04 smithi016 ceph-mon[87889]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 85 B/s wr, 195 op/s 2024-09-17T12:33:05.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:04 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:33:05.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:04 smithi016 ceph-mon[89784]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 85 B/s wr, 195 op/s 2024-09-17T12:33:05.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:04 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:33:06.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:33:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:33:06] "GET /metrics HTTP/1.1" 200 38719 "" "Prometheus/2.43.0" 2024-09-17T12:33:07.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:06 smithi196 ceph-mon[80339]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 170 B/s wr, 174 op/s 2024-09-17T12:33:07.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:06 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:33:07.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:06 smithi016 ceph-mon[87889]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 170 B/s wr, 174 op/s 2024-09-17T12:33:07.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:06 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:33:07.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:06 smithi016 ceph-mon[89784]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 170 B/s wr, 174 op/s 2024-09-17T12:33:07.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:06 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:33:08.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:07 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:08.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:07 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:08.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:07 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:33:08.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:07 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:08.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:07 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:08.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:07 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:33:08.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:07 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:08.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:07 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:08.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:07 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:33:09.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:08 smithi196 ceph-mon[80339]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 170 B/s wr, 194 op/s 2024-09-17T12:33:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:08 smithi016 ceph-mon[87889]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 170 B/s wr, 194 op/s 2024-09-17T12:33:09.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:08 smithi016 ceph-mon[89784]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 170 B/s wr, 194 op/s 2024-09-17T12:33:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:10 smithi016 ceph-mon[87889]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 170 B/s wr, 170 op/s 2024-09-17T12:33:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:10 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:10 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:10 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:11.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:10 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:10 smithi016 ceph-mon[89784]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 170 B/s wr, 170 op/s 2024-09-17T12:33:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:10 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:10 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:10 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:11.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:10 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:11.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:10 smithi196 ceph-mon[80339]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 170 B/s wr, 170 op/s 2024-09-17T12:33:11.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:10 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:11.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:10 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:11.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:10 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:11.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:10 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:12.433 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:12 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:12.433 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:12 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:12.433 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:12 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:12.433 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:12 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:12.433 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:12 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:12.433 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:12 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:12.433 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:12 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:12.434 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:12 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:12.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:12 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:12.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:12 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:12.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:12 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:12.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:12 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:13.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:13 smithi016 ceph-mon[87889]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 255 B/s wr, 176 op/s 2024-09-17T12:33:13.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:13 smithi016 ceph-mon[89784]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 255 B/s wr, 176 op/s 2024-09-17T12:33:13.514 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:13 smithi196 ceph-mon[80339]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 255 B/s wr, 176 op/s 2024-09-17T12:33:15.239 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-17T12:33:15.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:15.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:33:15.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:33:15.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: Upgrade: Setting container_image for all rgw 2024-09-17T12:33:15.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.tquwud"} : dispatch 2024-09-17T12:33:15.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.tquwud"} : dispatch 2024-09-17T12:33:15.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.tquwud"}]': finished 2024-09-17T12:33:15.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wvbocw"} : dispatch 2024-09-17T12:33:15.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wvbocw"} : dispatch 2024-09-17T12:33:15.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wvbocw"}]': finished 2024-09-17T12:33:15.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.qdkyfn"} : dispatch 2024-09-17T12:33:15.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.qdkyfn"} : dispatch 2024-09-17T12:33:15.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.qdkyfn"}]': finished 2024-09-17T12:33:15.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.ghtvuz"} : dispatch 2024-09-17T12:33:15.242 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-17T12:33:15.242 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.243 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.243 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.243 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.243 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:15.243 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:33:15.243 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.244 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:33:15.244 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.244 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.244 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.244 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.244 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.244 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.244 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: Upgrade: Setting container_image for all rgw 2024-09-17T12:33:15.244 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.244 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.tquwud"} : dispatch 2024-09-17T12:33:15.245 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.tquwud"} : dispatch 2024-09-17T12:33:15.245 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.tquwud"}]': finished 2024-09-17T12:33:15.245 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wvbocw"} : dispatch 2024-09-17T12:33:15.245 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wvbocw"} : dispatch 2024-09-17T12:33:15.245 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wvbocw"}]': finished 2024-09-17T12:33:15.245 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.qdkyfn"} : dispatch 2024-09-17T12:33:15.245 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.qdkyfn"} : dispatch 2024-09-17T12:33:15.245 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.qdkyfn"}]': finished 2024-09-17T12:33:15.245 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.ghtvuz"} : dispatch 2024-09-17T12:33:15.246 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.ghtvuz"} : dispatch 2024-09-17T12:33:15.246 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.ghtvuz"}]': finished 2024-09-17T12:33:15.246 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.246 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.246 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.246 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.247 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.247 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.ghtvuz"} : dispatch 2024-09-17T12:33:15.247 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.ghtvuz"}]': finished 2024-09-17T12:33:15.247 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.247 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.247 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.247 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.248 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:15 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-17T12:33:15.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:33:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:33:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: Upgrade: Setting container_image for all rgw 2024-09-17T12:33:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.tquwud"} : dispatch 2024-09-17T12:33:15.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.tquwud"} : dispatch 2024-09-17T12:33:15.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.tquwud"}]': finished 2024-09-17T12:33:15.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wvbocw"} : dispatch 2024-09-17T12:33:15.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wvbocw"} : dispatch 2024-09-17T12:33:15.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wvbocw"}]': finished 2024-09-17T12:33:15.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.qdkyfn"} : dispatch 2024-09-17T12:33:15.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.qdkyfn"} : dispatch 2024-09-17T12:33:15.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.qdkyfn"}]': finished 2024-09-17T12:33:15.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.ghtvuz"} : dispatch 2024-09-17T12:33:15.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.ghtvuz"} : dispatch 2024-09-17T12:33:15.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.ghtvuz"}]': finished 2024-09-17T12:33:15.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:15.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:15.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:15 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:16.276 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:16 smithi016 ceph-mon[87889]: Upgrade: Setting container_image for all rbd-mirror 2024-09-17T12:33:16.276 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:16 smithi016 ceph-mon[87889]: Upgrade: Setting container_image for all ceph-exporter 2024-09-17T12:33:16.276 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:16 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:16.276 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:16 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-17T12:33:16.276 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:16 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-17T12:33:16.276 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:16 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:16.276 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:33:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:33:16] "GET /metrics HTTP/1.1" 200 38719 "" "Prometheus/2.43.0" 2024-09-17T12:33:16.277 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:16 smithi016 ceph-mon[89784]: Upgrade: Setting container_image for all rbd-mirror 2024-09-17T12:33:16.277 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:16 smithi016 ceph-mon[89784]: Upgrade: Setting container_image for all ceph-exporter 2024-09-17T12:33:16.277 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:16 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:16.277 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:16 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-17T12:33:16.277 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:16 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-17T12:33:16.277 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:16 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:16.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:16 smithi196 ceph-mon[80339]: Upgrade: Setting container_image for all rbd-mirror 2024-09-17T12:33:16.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:16 smithi196 ceph-mon[80339]: Upgrade: Setting container_image for all ceph-exporter 2024-09-17T12:33:16.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:16 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:16.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:16 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-17T12:33:16.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:16 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.qaboca", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-17T12:33:16.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:16 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:17.427 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:17 smithi016 ceph-mon[87889]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 170 B/s wr, 122 op/s 2024-09-17T12:33:17.427 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:17 smithi016 ceph-mon[87889]: Upgrade: Updating iscsi.foo.smithi016.qaboca 2024-09-17T12:33:17.428 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:17 smithi016 ceph-mon[87889]: Deploying daemon iscsi.foo.smithi016.qaboca on smithi016 2024-09-17T12:33:17.428 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:17 smithi016 ceph-mon[89784]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 170 B/s wr, 122 op/s 2024-09-17T12:33:17.428 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:17 smithi016 ceph-mon[89784]: Upgrade: Updating iscsi.foo.smithi016.qaboca 2024-09-17T12:33:17.428 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:17 smithi016 ceph-mon[89784]: Deploying daemon iscsi.foo.smithi016.qaboca on smithi016 2024-09-17T12:33:17.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:17 smithi196 ceph-mon[80339]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 170 B/s wr, 122 op/s 2024-09-17T12:33:17.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:17 smithi196 ceph-mon[80339]: Upgrade: Updating iscsi.foo.smithi016.qaboca 2024-09-17T12:33:17.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:17 smithi196 ceph-mon[80339]: Deploying daemon iscsi.foo.smithi016.qaboca on smithi016 2024-09-17T12:33:18.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:18 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:33:18.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:18 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:33:18.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:18 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:33:19.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:19 smithi016 ceph-mon[87889]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 85 B/s wr, 96 op/s 2024-09-17T12:33:19.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:19 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:33:19.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:19 smithi016 ceph-mon[89784]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 85 B/s wr, 96 op/s 2024-09-17T12:33:19.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:19 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:33:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:19 smithi196 ceph-mon[80339]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 85 B/s wr, 96 op/s 2024-09-17T12:33:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:19 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:33:21.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:21 smithi016 ceph-mon[87889]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 85 B/s wr, 58 op/s 2024-09-17T12:33:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:21 smithi016 ceph-mon[89784]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 85 B/s wr, 58 op/s 2024-09-17T12:33:21.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:21 smithi196 ceph-mon[80339]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 85 B/s wr, 58 op/s 2024-09-17T12:33:23.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:23 smithi016 ceph-mon[87889]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 85 B/s wr, 40 op/s 2024-09-17T12:33:23.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:23 smithi016 ceph-mon[89784]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 85 B/s wr, 40 op/s 2024-09-17T12:33:23.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:23 smithi196 ceph-mon[80339]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 85 B/s wr, 40 op/s 2024-09-17T12:33:25.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:25 smithi016 ceph-mon[87889]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T12:33:25.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:25 smithi016 ceph-mon[89784]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T12:33:25.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:25 smithi196 ceph-mon[80339]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T12:33:26.192 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:33:26.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:33:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:33:26] "GET /metrics HTTP/1.1" 200 38773 "" "Prometheus/2.43.0" 2024-09-17T12:33:26.780 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:33:26.780 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (11m) 16s ago 12m 24.7M - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:33:26.780 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (12m) 16s ago 12m 88.0M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:33:26.780 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 running (6m) 16s ago 10m 62.0M - 3.8 980a0bfcd684 843db1407344 2024-09-17T12:33:26.780 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (6m) 16s ago 16m 456M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:33:26.780 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (6m) 16s ago 17m 525M - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:33:26.780 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (6m) 16s ago 18m 53.2M 2048M 19.3.0-4937-g74984932 6c1fcab85602 4a83b787c605 2024-09-17T12:33:26.780 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (5m) 16s ago 16m 41.6M 2048M 19.3.0-4937-g74984932 6c1fcab85602 dc48dfc9e40c 2024-09-17T12:33:26.780 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (5m) 16s ago 16m 45.1M 2048M 19.3.0-4937-g74984932 6c1fcab85602 aa5074acd22a 2024-09-17T12:33:26.780 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (11m) 16s ago 11m 15.1M - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (11m) 16s ago 11m 16.2M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (4m) 16s ago 15m 56.3M 2737M 19.3.0-4937-g74984932 6c1fcab85602 d12e6f26f048 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (4m) 16s ago 15m 64.8M 2737M 19.3.0-4937-g74984932 6c1fcab85602 cd2f6e2964dd 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (3m) 16s ago 15m 50.1M 2737M 19.3.0-4937-g74984932 6c1fcab85602 99430f24b824 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (3m) 16s ago 14m 54.4M 2737M 19.3.0-4937-g74984932 6c1fcab85602 1d4309778a0d 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (2m) 16s ago 14m 55.2M 2992M 19.3.0-4937-g74984932 6c1fcab85602 0c739e0cd187 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (118s) 16s ago 13m 51.2M 2992M 19.3.0-4937-g74984932 6c1fcab85602 71f93c731227 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (84s) 16s ago 13m 49.3M 2992M 19.3.0-4937-g74984932 6c1fcab85602 26b8884ca832 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (51s) 16s ago 13m 49.2M 2992M 19.3.0-4937-g74984932 6c1fcab85602 3ebdff7d6386 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (6m) 16s ago 12m 57.2M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (37s) 16s ago 10m 75.4M - 19.3.0-4937-g74984932 6c1fcab85602 41b68272c154 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (25s) 16s ago 11m 76.3M - 19.3.0-4937-g74984932 6c1fcab85602 43fd3378da10 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (31s) 16s ago 10m 76.5M - 19.3.0-4937-g74984932 6c1fcab85602 832c8d87d4a4 2024-09-17T12:33:26.781 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (20s) 16s ago 10m 75.2M - 19.3.0-4937-g74984932 6c1fcab85602 2420b80fda6a 2024-09-17T12:33:27.134 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:33:27.134 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:33:27.134 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:33:27.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:33:27.134 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:33:27.134 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:33:27.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:33:27.134 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:33:27.134 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 8 2024-09-17T12:33:27.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:33:27.135 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:33:27.135 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 4 2024-09-17T12:33:27.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:33:27.135 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:33:27.135 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 17 2024-09-17T12:33:27.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:33:27.135 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:33:27.438 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:33:27.438 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:33:27.438 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:33:27.438 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:33:27.438 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:33:27.438 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw", 2024-09-17T12:33:27.438 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-09-17T12:33:27.438 INFO:teuthology.orchestra.run.smithi016.stdout: "osd", 2024-09-17T12:33:27.438 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:33:27.439 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:33:27.439 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "17/23 daemons upgraded", 2024-09-17T12:33:27.439 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading iscsi daemons", 2024-09-17T12:33:27.439 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-09-17T12:33:27.439 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:33:27.439 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:27 smithi016 ceph-mon[87889]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:33:27.439 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:27 smithi016 ceph-mon[87889]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:27.439 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:27 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/3350538455' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:27.439 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:27 smithi016 ceph-mon[89784]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:33:27.440 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:27 smithi016 ceph-mon[89784]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:27.440 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:27 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/3350538455' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:27 smithi196 ceph-mon[80339]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:33:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:27 smithi196 ceph-mon[80339]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:27.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:27 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/3350538455' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:33:27.802 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-09-17T12:33:28.382 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:28 smithi016 ceph-mon[87889]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:33:28.382 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:28 smithi016 ceph-mon[87889]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:28.383 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:28 smithi016 ceph-mon[87889]: from='client.44594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:28.383 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:28 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/2094734433' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:33:28.383 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:28 smithi016 ceph-mon[89784]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:33:28.383 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:28 smithi016 ceph-mon[89784]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:28.383 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:28 smithi016 ceph-mon[89784]: from='client.44594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:28.383 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:28 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/2094734433' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:33:28.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:28 smithi196 ceph-mon[80339]: from='client.25262 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:33:28.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:28 smithi196 ceph-mon[80339]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:28.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:28 smithi196 ceph-mon[80339]: from='client.44594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:28.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:28 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/2094734433' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:33:29.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:33:29 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:33:29.130+0000 7f9d29bd7640 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.qaboca on host smithi016 failed. 2024-09-17T12:33:29.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:29 smithi016 ceph-mon[87889]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:29.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:29 smithi016 ceph-mon[87889]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:33:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:29 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:29.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:29 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:33:29.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:29 smithi016 ceph-mon[89784]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:29.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:29 smithi016 ceph-mon[89784]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:33:29.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:29 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:29.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:29 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:33:29.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:29 smithi196 ceph-mon[80339]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:29 smithi196 ceph-mon[80339]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:33:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:29 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:29 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:33:30.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:30 smithi016 ceph-mon[87889]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.qaboca on host smithi016 failed. 2024-09-17T12:33:30.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:30 smithi016 ceph-mon[87889]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-09-17T12:33:30.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:30 smithi016 ceph-mon[87889]: Health check failed: Upgrading daemon iscsi.foo.smithi016.qaboca on host smithi016 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-09-17T12:33:30.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:30 smithi016 ceph-mon[89784]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.qaboca on host smithi016 failed. 2024-09-17T12:33:30.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:30 smithi016 ceph-mon[89784]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-09-17T12:33:30.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:30 smithi016 ceph-mon[89784]: Health check failed: Upgrading daemon iscsi.foo.smithi016.qaboca on host smithi016 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-09-17T12:33:30.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:30 smithi196 ceph-mon[80339]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.qaboca on host smithi016 failed. 2024-09-17T12:33:30.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:30 smithi196 ceph-mon[80339]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-09-17T12:33:30.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:30 smithi196 ceph-mon[80339]: Health check failed: Upgrading daemon iscsi.foo.smithi016.qaboca on host smithi016 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-09-17T12:33:32.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:32 smithi016 ceph-mon[87889]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-09-17T12:33:32.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:32 smithi016 ceph-mon[89784]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-09-17T12:33:32.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:32 smithi196 ceph-mon[80339]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-09-17T12:33:34.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:34.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:33:34.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[87889]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-09-17T12:33:34.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[87889]: Checking dashboard <-> RGW credentials 2024-09-17T12:33:34.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:33:34.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:34.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:33:34.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:34.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:33:34.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[89784]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-09-17T12:33:34.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[89784]: Checking dashboard <-> RGW credentials 2024-09-17T12:33:34.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:33:34.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:34.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:33:34.193 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:33 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:33 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:33 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:33 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:33 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:33 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:33 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:33:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:33 smithi196 ceph-mon[80339]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-09-17T12:33:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:33 smithi196 ceph-mon[80339]: Checking dashboard <-> RGW credentials 2024-09-17T12:33:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:33 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:33 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:33:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:33 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:33:34.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:33 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:33:34.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:33 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:34 smithi016 ceph-mon[87889]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-09-17T12:33:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:34 smithi016 ceph-mon[87889]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-09-17T12:33:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:34 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:33:35.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:34 smithi016 ceph-mon[87889]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T12:33:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:34 smithi016 ceph-mon[89784]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-09-17T12:33:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:34 smithi016 ceph-mon[89784]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-09-17T12:33:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:34 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:34 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:33:35.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:34 smithi016 ceph-mon[89784]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T12:33:35.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:34 smithi196 ceph-mon[80339]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-09-17T12:33:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:34 smithi196 ceph-mon[80339]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-09-17T12:33:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:34 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:34 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:33:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:34 smithi196 ceph-mon[80339]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T12:33:36.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:33:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:33:36] "GET /metrics HTTP/1.1" 200 38852 "" "Prometheus/2.43.0" 2024-09-17T12:33:37.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:36 smithi016 ceph-mon[87889]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-17T12:33:37.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:36 smithi016 ceph-mon[89784]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-17T12:33:37.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:36 smithi196 ceph-mon[80339]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-17T12:33:39.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:38 smithi016 ceph-mon[87889]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-09-17T12:33:39.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:38 smithi016 ceph-mon[89784]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-09-17T12:33:39.252 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:38 smithi196 ceph-mon[80339]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-09-17T12:33:41.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:40 smithi016 ceph-mon[87889]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 2 op/s 2024-09-17T12:33:41.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:40 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/4031437005' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-17T12:33:41.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:40 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/784817466' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2617309680"} : dispatch 2024-09-17T12:33:41.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:40 smithi016 ceph-mon[89784]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 2 op/s 2024-09-17T12:33:41.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:40 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/4031437005' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-17T12:33:41.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:40 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/784817466' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2617309680"} : dispatch 2024-09-17T12:33:41.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:40 smithi196 ceph-mon[80339]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 2 op/s 2024-09-17T12:33:41.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:40 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/4031437005' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-17T12:33:41.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:40 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/784817466' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2617309680"} : dispatch 2024-09-17T12:33:42.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:41 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/784817466' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2617309680"}]': finished 2024-09-17T12:33:42.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:41 smithi196 ceph-mon[80339]: osdmap e146: 8 total, 8 up, 8 in 2024-09-17T12:33:42.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:41 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/1619056976' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4015930328"} : dispatch 2024-09-17T12:33:42.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:41 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/784817466' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2617309680"}]': finished 2024-09-17T12:33:42.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:41 smithi016 ceph-mon[87889]: osdmap e146: 8 total, 8 up, 8 in 2024-09-17T12:33:42.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:41 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/1619056976' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4015930328"} : dispatch 2024-09-17T12:33:42.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:41 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/784817466' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2617309680"}]': finished 2024-09-17T12:33:42.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:41 smithi016 ceph-mon[89784]: osdmap e146: 8 total, 8 up, 8 in 2024-09-17T12:33:42.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:41 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/1619056976' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4015930328"} : dispatch 2024-09-17T12:33:43.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:42 smithi196 ceph-mon[80339]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 2 op/s 2024-09-17T12:33:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:42 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/1619056976' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4015930328"}]': finished 2024-09-17T12:33:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:42 smithi196 ceph-mon[80339]: osdmap e147: 8 total, 8 up, 8 in 2024-09-17T12:33:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:42 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/888654044' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4015930328"} : dispatch 2024-09-17T12:33:43.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:42 smithi196 ceph-mon[80339]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4015930328"} : dispatch 2024-09-17T12:33:43.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:42 smithi016 ceph-mon[87889]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 2 op/s 2024-09-17T12:33:43.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:42 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/1619056976' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4015930328"}]': finished 2024-09-17T12:33:43.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:42 smithi016 ceph-mon[87889]: osdmap e147: 8 total, 8 up, 8 in 2024-09-17T12:33:43.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:42 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/888654044' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4015930328"} : dispatch 2024-09-17T12:33:43.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:42 smithi016 ceph-mon[87889]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4015930328"} : dispatch 2024-09-17T12:33:43.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:42 smithi016 ceph-mon[89784]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 2 op/s 2024-09-17T12:33:43.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:42 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/1619056976' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4015930328"}]': finished 2024-09-17T12:33:43.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:42 smithi016 ceph-mon[89784]: osdmap e147: 8 total, 8 up, 8 in 2024-09-17T12:33:43.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:42 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/888654044' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4015930328"} : dispatch 2024-09-17T12:33:43.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:42 smithi016 ceph-mon[89784]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4015930328"} : dispatch 2024-09-17T12:33:44.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:43 smithi196 ceph-mon[80339]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4015930328"}]': finished 2024-09-17T12:33:44.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:43 smithi196 ceph-mon[80339]: osdmap e148: 8 total, 8 up, 8 in 2024-09-17T12:33:44.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:43 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/3541258871' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/49334351"} : dispatch 2024-09-17T12:33:44.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:43 smithi016 ceph-mon[87889]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4015930328"}]': finished 2024-09-17T12:33:44.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:43 smithi016 ceph-mon[87889]: osdmap e148: 8 total, 8 up, 8 in 2024-09-17T12:33:44.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:43 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/3541258871' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/49334351"} : dispatch 2024-09-17T12:33:44.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:43 smithi016 ceph-mon[89784]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4015930328"}]': finished 2024-09-17T12:33:44.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:43 smithi016 ceph-mon[89784]: osdmap e148: 8 total, 8 up, 8 in 2024-09-17T12:33:44.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:43 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/3541258871' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/49334351"} : dispatch 2024-09-17T12:33:45.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:44 smithi196 ceph-mon[80339]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T12:33:45.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:44 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/3541258871' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/49334351"}]': finished 2024-09-17T12:33:45.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:44 smithi196 ceph-mon[80339]: osdmap e149: 8 total, 8 up, 8 in 2024-09-17T12:33:45.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:44 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/2236891374' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3570443482"} : dispatch 2024-09-17T12:33:45.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:44 smithi196 ceph-mon[80339]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3570443482"} : dispatch 2024-09-17T12:33:45.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:44 smithi016 ceph-mon[89784]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T12:33:45.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:44 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/3541258871' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/49334351"}]': finished 2024-09-17T12:33:45.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:44 smithi016 ceph-mon[89784]: osdmap e149: 8 total, 8 up, 8 in 2024-09-17T12:33:45.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:44 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/2236891374' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3570443482"} : dispatch 2024-09-17T12:33:45.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:44 smithi016 ceph-mon[89784]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3570443482"} : dispatch 2024-09-17T12:33:45.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:44 smithi016 ceph-mon[87889]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T12:33:45.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:44 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/3541258871' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/49334351"}]': finished 2024-09-17T12:33:45.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:44 smithi016 ceph-mon[87889]: osdmap e149: 8 total, 8 up, 8 in 2024-09-17T12:33:45.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:44 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/2236891374' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3570443482"} : dispatch 2024-09-17T12:33:45.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:44 smithi016 ceph-mon[87889]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3570443482"} : dispatch 2024-09-17T12:33:46.217 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:45 smithi016 ceph-mon[87889]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3570443482"}]': finished 2024-09-17T12:33:46.217 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:45 smithi016 ceph-mon[87889]: osdmap e150: 8 total, 8 up, 8 in 2024-09-17T12:33:46.217 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:45 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/374901699' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3622675476"} : dispatch 2024-09-17T12:33:46.217 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:45 smithi016 ceph-mon[87889]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3622675476"} : dispatch 2024-09-17T12:33:46.218 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:45 smithi016 ceph-mon[89784]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3570443482"}]': finished 2024-09-17T12:33:46.218 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:45 smithi016 ceph-mon[89784]: osdmap e150: 8 total, 8 up, 8 in 2024-09-17T12:33:46.218 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:45 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/374901699' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3622675476"} : dispatch 2024-09-17T12:33:46.218 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:45 smithi016 ceph-mon[89784]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3622675476"} : dispatch 2024-09-17T12:33:46.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:45 smithi196 ceph-mon[80339]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3570443482"}]': finished 2024-09-17T12:33:46.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:45 smithi196 ceph-mon[80339]: osdmap e150: 8 total, 8 up, 8 in 2024-09-17T12:33:46.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:45 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/374901699' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3622675476"} : dispatch 2024-09-17T12:33:46.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:45 smithi196 ceph-mon[80339]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd={"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3622675476"} : dispatch 2024-09-17T12:33:46.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:33:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:33:46] "GET /metrics HTTP/1.1" 200 38852 "" "Prometheus/2.43.0" 2024-09-17T12:33:47.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:46 smithi016 ceph-mon[87889]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T12:33:47.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:46 smithi016 ceph-mon[87889]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3622675476"}]': finished 2024-09-17T12:33:47.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:46 smithi016 ceph-mon[87889]: osdmap e151: 8 total, 8 up, 8 in 2024-09-17T12:33:47.235 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:46 smithi016 ceph-mon[89784]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T12:33:47.235 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:46 smithi016 ceph-mon[89784]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3622675476"}]': finished 2024-09-17T12:33:47.235 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:46 smithi016 ceph-mon[89784]: osdmap e151: 8 total, 8 up, 8 in 2024-09-17T12:33:47.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:46 smithi196 ceph-mon[80339]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T12:33:47.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:46 smithi196 ceph-mon[80339]: from='client.? ' entity='client.iscsi.foo.smithi016.qaboca' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3622675476"}]': finished 2024-09-17T12:33:47.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:46 smithi196 ceph-mon[80339]: osdmap e151: 8 total, 8 up, 8 in 2024-09-17T12:33:49.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:48 smithi196 ceph-mon[80339]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:33:49.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:48 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:49.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:48 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:33:49.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:48 smithi016 ceph-mon[87889]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:33:49.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:48 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:49.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:48 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:33:49.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:48 smithi016 ceph-mon[89784]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:33:49.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:48 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:33:49.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:48 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:33:51.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:50 smithi196 ceph-mon[80339]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:33:51.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:50 smithi196 ceph-mon[80339]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:33:51.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:50 smithi016 ceph-mon[87889]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:33:51.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:50 smithi016 ceph-mon[87889]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:33:51.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:50 smithi016 ceph-mon[89784]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:33:51.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:50 smithi016 ceph-mon[89784]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:33:53.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:52 smithi196 ceph-mon[80339]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-09-17T12:33:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:52 smithi016 ceph-mon[87889]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-09-17T12:33:53.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:52 smithi016 ceph-mon[89784]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-09-17T12:33:55.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:54 smithi196 ceph-mon[80339]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:33:55.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:54 smithi016 ceph-mon[87889]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:33:55.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:54 smithi016 ceph-mon[89784]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:33:56.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:33:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:33:56] "GET /metrics HTTP/1.1" 200 38850 "" "Prometheus/2.43.0" 2024-09-17T12:33:57.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:56 smithi196 ceph-mon[80339]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:33:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:56 smithi016 ceph-mon[87889]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:33:57.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:56 smithi016 ceph-mon[89784]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:33:58.129 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:33:58.426 INFO:teuthology.orchestra.run.smithi016.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.qaboca on host smithi016 failed." 2024-09-17T12:33:59.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:58 smithi196 ceph-mon[80339]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T12:33:59.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:58 smithi196 ceph-mon[80339]: from='client.44714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:59.421 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:58 smithi016 ceph-mon[87889]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T12:33:59.421 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:58 smithi016 ceph-mon[87889]: from='client.44714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:59.421 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:58 smithi016 ceph-mon[89784]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T12:33:59.421 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:58 smithi016 ceph-mon[89784]: from='client.44714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:33:59.447 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph orch ps' 2024-09-17T12:33:59.662 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:34:00.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:33:59 smithi016 ceph-mon[87889]: from='client.44720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:34:00.091 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:33:59 smithi016 ceph-mon[89784]: from='client.44720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:34:00.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:33:59 smithi196 ceph-mon[80339]: from='client.44720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:34:01.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:00 smithi196 ceph-mon[80339]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:01.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:00 smithi196 ceph-mon[80339]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:01.381 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:34:01.382 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (12m) 28s ago 12m - - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:34:01.382 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (12m) 28s ago 12m 88.0M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:34:01.382 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 unknown 28s ago 11m - - 2024-09-17T12:34:01.382 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (6m) 28s ago 16m 456M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:34:01.382 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (7m) 28s ago 18m - - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:34:01.382 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (6m) 28s ago 18m - 2048M 19.3.0-4937-g74984932 6c1fcab85602 4a83b787c605 2024-09-17T12:34:01.382 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (6m) 28s ago 17m 44.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 dc48dfc9e40c 2024-09-17T12:34:01.382 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (6m) 28s ago 17m - 2048M 19.3.0-4937-g74984932 6c1fcab85602 aa5074acd22a 2024-09-17T12:34:01.382 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (12m) 28s ago 12m - - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:34:01.382 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (12m) 28s ago 12m 15.9M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:34:01.382 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (5m) 28s ago 16m - 2737M 19.3.0-4937-g74984932 6c1fcab85602 d12e6f26f048 2024-09-17T12:34:01.382 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (4m) 28s ago 16m - 2737M 19.3.0-4937-g74984932 6c1fcab85602 cd2f6e2964dd 2024-09-17T12:34:01.383 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (4m) 28s ago 15m - 2737M 19.3.0-4937-g74984932 6c1fcab85602 99430f24b824 2024-09-17T12:34:01.383 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (3m) 28s ago 15m - 2737M 19.3.0-4937-g74984932 6c1fcab85602 1d4309778a0d 2024-09-17T12:34:01.383 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (3m) 28s ago 14m 54.7M 2992M 19.3.0-4937-g74984932 6c1fcab85602 0c739e0cd187 2024-09-17T12:34:01.383 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (2m) 28s ago 14m 50.4M 2992M 19.3.0-4937-g74984932 6c1fcab85602 71f93c731227 2024-09-17T12:34:01.383 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (119s) 28s ago 14m 49.0M 2992M 19.3.0-4937-g74984932 6c1fcab85602 26b8884ca832 2024-09-17T12:34:01.383 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (86s) 28s ago 13m 49.6M 2992M 19.3.0-4937-g74984932 6c1fcab85602 3ebdff7d6386 2024-09-17T12:34:01.383 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (6m) 28s ago 13m 57.2M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:34:01.383 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (71s) 28s ago 11m - - 19.3.0-4937-g74984932 6c1fcab85602 41b68272c154 2024-09-17T12:34:01.383 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (60s) 28s ago 11m 76.5M - 19.3.0-4937-g74984932 6c1fcab85602 43fd3378da10 2024-09-17T12:34:01.383 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (66s) 28s ago 11m - - 19.3.0-4937-g74984932 6c1fcab85602 832c8d87d4a4 2024-09-17T12:34:01.383 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (54s) 28s ago 11m 75.4M - 19.3.0-4937-g74984932 6c1fcab85602 2420b80fda6a 2024-09-17T12:34:01.384 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:00 smithi016 ceph-mon[87889]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:01.384 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:00 smithi016 ceph-mon[87889]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:01.384 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:00 smithi016 ceph-mon[89784]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:01.384 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:00 smithi016 ceph-mon[89784]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:02.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph versions' 2024-09-17T12:34:02.267 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:34:03.270 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:02 smithi016 ceph-mon[89784]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:34:03.270 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:02 smithi016 ceph-mon[89784]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:03.270 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:02 smithi016 ceph-mon[87889]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:34:03.270 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:02 smithi016 ceph-mon[87889]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:03.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:02 smithi196 ceph-mon[80339]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:34:03.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:02 smithi196 ceph-mon[80339]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:04.070 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:34:04.071 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:34:04.071 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:34:04.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:34:04.071 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:34:04.071 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:34:04.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:34:04.071 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:34:04.071 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 8 2024-09-17T12:34:04.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:34:04.071 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:34:04.071 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 4 2024-09-17T12:34:04.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:34:04.072 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:34:04.072 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 17 2024-09-17T12:34:04.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:34:04.072 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:34:04.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:03 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:34:04.433 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:03 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:34:04.433 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:03 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:34:04.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-09-17T12:34:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:34:05.222 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:04 smithi016 ceph-mon[87889]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:05.222 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:04 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/3118136994' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:34:05.222 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:04 smithi016 ceph-mon[89784]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:05.223 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:04 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/3118136994' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:34:05.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:04 smithi196 ceph-mon[80339]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:05.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:04 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/3118136994' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:34:06.429 INFO:teuthology.orchestra.run.smithi016.stdout:wait for servicemap items w/ changing names to refresh 2024-09-17T12:34:06.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:34:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:34:06] "GET /metrics HTTP/1.1" 200 38847 "" "Prometheus/2.43.0" 2024-09-17T12:34:07.071 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'sleep 60' 2024-09-17T12:34:07.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:06 smithi196 ceph-mon[80339]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:07.287 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:34:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:06 smithi016 ceph-mon[87889]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:07.408 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:06 smithi016 ceph-mon[89784]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:09.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:09 smithi196 ceph-mon[80339]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:09.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:09 smithi016 ceph-mon[87889]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:09.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:09 smithi016 ceph-mon[89784]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:11.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:11 smithi196 ceph-mon[80339]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:11.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:11 smithi196 ceph-mon[80339]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:11.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:11 smithi016 ceph-mon[87889]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:11.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:11 smithi016 ceph-mon[87889]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:11.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:11 smithi016 ceph-mon[89784]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:11.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:11 smithi016 ceph-mon[89784]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:13.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:13 smithi196 ceph-mon[80339]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:13.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:13 smithi016 ceph-mon[87889]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:13.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:13 smithi016 ceph-mon[89784]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:15.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:15 smithi196 ceph-mon[80339]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:15.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:15 smithi016 ceph-mon[87889]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:15.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:15 smithi016 ceph-mon[89784]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:16.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:34:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:34:16] "GET /metrics HTTP/1.1" 200 38847 "" "Prometheus/2.43.0" 2024-09-17T12:34:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:17 smithi196 ceph-mon[80339]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:17.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:17 smithi016 ceph-mon[87889]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:17.357 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:17 smithi016 ceph-mon[89784]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:19.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:19 smithi196 ceph-mon[80339]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:19 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:34:19.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:19 smithi016 ceph-mon[89784]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:19.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:19 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:34:19.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:19 smithi016 ceph-mon[87889]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:19.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:19 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:34:21.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:21 smithi196 ceph-mon[80339]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:21.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:21 smithi196 ceph-mon[80339]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:21.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:21 smithi016 ceph-mon[87889]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:21.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:21 smithi016 ceph-mon[87889]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:21 smithi016 ceph-mon[89784]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:21.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:21 smithi016 ceph-mon[89784]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:23.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:23 smithi196 ceph-mon[80339]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:23.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:23 smithi016 ceph-mon[87889]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:23.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:23 smithi016 ceph-mon[89784]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:25.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:25 smithi196 ceph-mon[80339]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:25.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:25 smithi016 ceph-mon[87889]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:25.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:25 smithi016 ceph-mon[89784]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:26.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:34:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:34:26] "GET /metrics HTTP/1.1" 200 38848 "" "Prometheus/2.43.0" 2024-09-17T12:34:27.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:27 smithi016 ceph-mon[87889]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:27.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:27 smithi016 ceph-mon[89784]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:27 smithi196 ceph-mon[80339]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:29.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:29 smithi016 ceph-mon[87889]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:29.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:29 smithi016 ceph-mon[89784]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:29.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:29 smithi196 ceph-mon[80339]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:31.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:31 smithi016 ceph-mon[87889]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:31.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:31 smithi016 ceph-mon[87889]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:31.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:31 smithi016 ceph-mon[89784]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:31.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:31 smithi016 ceph-mon[89784]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:31.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:31 smithi196 ceph-mon[80339]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:31.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:31 smithi196 ceph-mon[80339]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:33.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:33 smithi016 ceph-mon[87889]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:33.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:33 smithi016 ceph-mon[89784]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:33.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:33 smithi196 ceph-mon[80339]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:34.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:34:34.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:34:34.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:34:34.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:34 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:34:34.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:34:34.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:34 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:34:34.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:34 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:34:34.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:34 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:34:34.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:34 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:34:34.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:34 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:34:34.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:34 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:34:34.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:34 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:34:34.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:34 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:34:34.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:34 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:34:34.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:34 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:34:35.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:35 smithi016 ceph-mon[87889]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:35.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:35 smithi016 ceph-mon[87889]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 981 B/s rd, 0 op/s 2024-09-17T12:34:35.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:35 smithi016 ceph-mon[87889]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:35.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:35 smithi016 ceph-mon[89784]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:35.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:35 smithi016 ceph-mon[89784]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 981 B/s rd, 0 op/s 2024-09-17T12:34:35.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:35 smithi016 ceph-mon[89784]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:35.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:35 smithi196 ceph-mon[80339]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:35.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:35 smithi196 ceph-mon[80339]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 981 B/s rd, 0 op/s 2024-09-17T12:34:35.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:35 smithi196 ceph-mon[80339]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:36.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:34:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:34:36] "GET /metrics HTTP/1.1" 200 38851 "" "Prometheus/2.43.0" 2024-09-17T12:34:37.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:37 smithi196 ceph-mon[80339]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-09-17T12:34:37.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:37 smithi016 ceph-mon[87889]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-09-17T12:34:37.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:37 smithi016 ceph-mon[89784]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-09-17T12:34:39.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:39 smithi016 ceph-mon[87889]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:39.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:39 smithi016 ceph-mon[89784]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:39.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:39 smithi196 ceph-mon[80339]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:41.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:41 smithi016 ceph-mon[87889]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:41.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:41 smithi016 ceph-mon[87889]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:41.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:41 smithi016 ceph-mon[89784]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:41.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:41 smithi016 ceph-mon[89784]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:41.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:41 smithi196 ceph-mon[80339]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:41.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:41 smithi196 ceph-mon[80339]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:43.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:43 smithi016 ceph-mon[87889]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-09-17T12:34:43.445 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:43 smithi016 ceph-mon[89784]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-09-17T12:34:43.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:43 smithi196 ceph-mon[80339]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-09-17T12:34:45.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:45 smithi016 ceph-mon[87889]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:34:45.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:45 smithi016 ceph-mon[89784]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:34:45.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:45 smithi196 ceph-mon[80339]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T12:34:46.593 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:34:46 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:34:46] "GET /metrics HTTP/1.1" 200 38851 "" "Prometheus/2.43.0" 2024-09-17T12:34:47.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:47 smithi016 ceph-mon[87889]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:47.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:47 smithi016 ceph-mon[89784]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:47.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:47 smithi196 ceph-mon[80339]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:49.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:49 smithi016 ceph-mon[89784]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:49.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:49 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:34:49.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:49 smithi016 ceph-mon[87889]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:49.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:49 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:34:49.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:49 smithi196 ceph-mon[80339]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:49.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:49 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:34:51.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:51 smithi016 ceph-mon[87889]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:51.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:51 smithi016 ceph-mon[87889]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:51.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:51 smithi016 ceph-mon[89784]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:51.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:51 smithi016 ceph-mon[89784]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:51.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:51 smithi196 ceph-mon[80339]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:51.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:51 smithi196 ceph-mon[80339]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:34:53.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:53 smithi016 ceph-mon[87889]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:53.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:53 smithi016 ceph-mon[89784]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:53.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:53 smithi196 ceph-mon[80339]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:55.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:55 smithi016 ceph-mon[87889]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:55.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:55 smithi016 ceph-mon[89784]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:55.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:55 smithi196 ceph-mon[80339]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:56.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:34:56 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:34:56] "GET /metrics HTTP/1.1" 200 38849 "" "Prometheus/2.43.0" 2024-09-17T12:34:57.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:57 smithi016 ceph-mon[87889]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:57.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:57 smithi016 ceph-mon[89784]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:57.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:57 smithi196 ceph-mon[80339]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:34:59.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:34:59 smithi016 ceph-mon[89784]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:59.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:34:59 smithi016 ceph-mon[87889]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:34:59.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:34:59 smithi196 ceph-mon[80339]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:01.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:01 smithi016 ceph-mon[87889]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:01.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:01 smithi016 ceph-mon[87889]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:35:01.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:01 smithi016 ceph-mon[89784]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:01.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:01 smithi016 ceph-mon[89784]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:35:01.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:01 smithi196 ceph-mon[80339]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:01.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:01 smithi196 ceph-mon[80339]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:35:03.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:03 smithi016 ceph-mon[87889]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:03.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:03 smithi016 ceph-mon[89784]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:03.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:03 smithi196 ceph-mon[80339]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:04.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:04 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:35:04.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:04 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:35:04.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:04 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:35:05.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:05 smithi016 ceph-mon[87889]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:05.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:05 smithi016 ceph-mon[89784]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:05.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:05 smithi196 ceph-mon[80339]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:06.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:06 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:35:06] "GET /metrics HTTP/1.1" 200 38847 "" "Prometheus/2.43.0" 2024-09-17T12:35:07.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:07 smithi016 ceph-mon[89784]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:07.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:07 smithi016 ceph-mon[87889]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:07.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:07 smithi196 ceph-mon[80339]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:09.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:09 smithi016 ceph-mon[89784]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:09.165 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:09 smithi016 ceph-mon[87889]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:09.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:09 smithi196 ceph-mon[80339]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:09.651 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph orch ps' 2024-09-17T12:35:09.869 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:35:11.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:11 smithi016 ceph-mon[87889]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:11.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:11 smithi016 ceph-mon[87889]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:35:11.400 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:11 smithi016 ceph-mon[89784]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:11.400 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:11 smithi016 ceph-mon[89784]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:35:11.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:11 smithi196 ceph-mon[80339]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:11.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:11 smithi196 ceph-mon[80339]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:35:11.613 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T12:35:11.613 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (13m) 98s ago 14m - - 0.25.0 c8568f914cd2 5103666cca2e 2024-09-17T12:35:11.614 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (13m) 98s ago 13m 88.0M - 9.4.7 954c08fa6188 52809a2352ad 2024-09-17T12:35:11.614 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.qaboca smithi016 *:5000 unknown 98s ago 12m - - 2024-09-17T12:35:11.614 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (7m) 98s ago 17m 456M - 19.3.0-4937-g74984932 6c1fcab85602 89153031794c 2024-09-17T12:35:11.614 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (8m) 98s ago 19m - - 19.3.0-4937-g74984932 6c1fcab85602 953d9c4b1666 2024-09-17T12:35:11.614 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (7m) 98s ago 19m - 2048M 19.3.0-4937-g74984932 6c1fcab85602 4a83b787c605 2024-09-17T12:35:11.614 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (7m) 98s ago 18m 44.8M 2048M 19.3.0-4937-g74984932 6c1fcab85602 dc48dfc9e40c 2024-09-17T12:35:11.614 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (7m) 98s ago 18m - 2048M 19.3.0-4937-g74984932 6c1fcab85602 aa5074acd22a 2024-09-17T12:35:11.614 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (13m) 98s ago 13m - - 1.5.0 0da6a335fe13 c74c1d1d2736 2024-09-17T12:35:11.614 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (13m) 98s ago 13m 15.9M - 1.5.0 0da6a335fe13 eaddaabd4509 2024-09-17T12:35:11.614 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (6m) 98s ago 17m - 2737M 19.3.0-4937-g74984932 6c1fcab85602 d12e6f26f048 2024-09-17T12:35:11.614 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (5m) 98s ago 17m - 2737M 19.3.0-4937-g74984932 6c1fcab85602 cd2f6e2964dd 2024-09-17T12:35:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (5m) 98s ago 16m - 2737M 19.3.0-4937-g74984932 6c1fcab85602 99430f24b824 2024-09-17T12:35:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (4m) 98s ago 16m - 2737M 19.3.0-4937-g74984932 6c1fcab85602 1d4309778a0d 2024-09-17T12:35:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (4m) 98s ago 16m 54.7M 2992M 19.3.0-4937-g74984932 6c1fcab85602 0c739e0cd187 2024-09-17T12:35:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (3m) 98s ago 15m 50.4M 2992M 19.3.0-4937-g74984932 6c1fcab85602 71f93c731227 2024-09-17T12:35:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (3m) 98s ago 15m 49.0M 2992M 19.3.0-4937-g74984932 6c1fcab85602 26b8884ca832 2024-09-17T12:35:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (2m) 98s ago 14m 49.6M 2992M 19.3.0-4937-g74984932 6c1fcab85602 3ebdff7d6386 2024-09-17T12:35:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (8m) 98s ago 14m 57.2M - 2.43.0 a07b618ecd1d fd41eb792eeb 2024-09-17T12:35:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.tquwud smithi016 *:8000 running (2m) 98s ago 12m - - 19.3.0-4937-g74984932 6c1fcab85602 41b68272c154 2024-09-17T12:35:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wvbocw smithi196 *:8000 running (2m) 98s ago 12m 76.5M - 19.3.0-4937-g74984932 6c1fcab85602 43fd3378da10 2024-09-17T12:35:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.qdkyfn smithi016 *:80 running (2m) 98s ago 12m - - 19.3.0-4937-g74984932 6c1fcab85602 832c8d87d4a4 2024-09-17T12:35:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.ghtvuz smithi196 *:80 running (2m) 98s ago 12m 75.4M - 19.3.0-4937-g74984932 6c1fcab85602 2420b80fda6a 2024-09-17T12:35:12.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph versions' 2024-09-17T12:35:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:35:13.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:13 smithi016 ceph-mon[87889]: from='client.44738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:13.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:13 smithi016 ceph-mon[87889]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:13.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:13 smithi016 ceph-mon[89784]: from='client.44738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:13.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:13 smithi016 ceph-mon[89784]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:13.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:13 smithi196 ceph-mon[80339]: from='client.44738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:13.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:13 smithi196 ceph-mon[80339]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:14.308 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:35:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-09-17T12:35:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 3 2024-09-17T12:35:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:35:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-09-17T12:35:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 2 2024-09-17T12:35:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:35:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-09-17T12:35:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 8 2024-09-17T12:35:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:35:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-09-17T12:35:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 4 2024-09-17T12:35:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-17T12:35:14.310 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-09-17T12:35:14.310 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)": 17 2024-09-17T12:35:14.310 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-17T12:35:14.310 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:35:15.014 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph orch upgrade status' 2024-09-17T12:35:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:35:15.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:15 smithi016 ceph-mon[87889]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:15.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:15 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/3704274573' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:35:15.353 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:15 smithi016 ceph-mon[89784]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:15.354 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:15 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/3704274573' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:35:15.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:15 smithi196 ceph-mon[80339]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:15.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:15 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/3704274573' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:35:16.319 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:16 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:35:16] "GET /metrics HTTP/1.1" 200 38847 "" "Prometheus/2.43.0" 2024-09-17T12:35:17.032 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:35:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb", 2024-09-17T12:35:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-09-17T12:35:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T12:35:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-09-17T12:35:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw", 2024-09-17T12:35:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-09-17T12:35:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "osd", 2024-09-17T12:35:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-09-17T12:35:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-17T12:35:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "17/23 daemons upgraded", 2024-09-17T12:35:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.qaboca on host smithi016 failed.", 2024-09-17T12:35:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": true 2024-09-17T12:35:17.033 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:35:17.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:17 smithi016 ceph-mon[89784]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:17.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:17 smithi016 ceph-mon[87889]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:17.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:17 smithi196 ceph-mon[80339]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:17.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph health detail' 2024-09-17T12:35:17.976 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:35:18.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:18 smithi016 ceph-mon[87889]: from='client.34809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:18.356 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:18 smithi016 ceph-mon[89784]: from='client.34809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:18.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:18 smithi196 ceph-mon[80339]: from='client.34809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:19.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:19 smithi016 ceph-mon[87889]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:19.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:19 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:35:19.333 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:19 smithi016 ceph-mon[89784]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:19.333 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:19 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:35:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:19 smithi196 ceph-mon[80339]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:19 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:35:19.818 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi016.qaboca on host smithi016 failed. 2024-09-17T12:35:19.818 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T12:35:19.818 INFO:teuthology.orchestra.run.smithi016.stdout: daemon iscsi.foo.smithi016.qaboca on smithi016 is in unknown state 2024-09-17T12:35:19.819 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.qaboca on host smithi016 failed. 2024-09-17T12:35:19.819 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrade daemon: iscsi.foo.smithi016.qaboca: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi016.qaboca ... 2024-09-17T12:35:19.819 INFO:teuthology.orchestra.run.smithi016.stdout:Creating ceph-iscsi config... 2024-09-17T12:35:19.819 INFO:teuthology.orchestra.run.smithi016.stdout:Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/iscsi-gateway.cfg 2024-09-17T12:35:19.819 INFO:teuthology.orchestra.run.smithi016.stdout:Write file: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/iscsi.foo.smithi016.qaboca/tcmu-runner-entrypoint.sh 2024-09-17T12:35:19.819 INFO:teuthology.orchestra.run.smithi016.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d5b9f4610\x2d74ee\x2d11ef\x2dbceb\x2dc7b262605968.slice/ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service 2024-09-17T12:35:19.819 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl start ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca 2024-09-17T12:35:19.819 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:35:19.819 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:35:19.819 INFO:teuthology.orchestra.run.smithi016.stdout:Traceback (most recent call last): 2024-09-17T12:35:19.819 INFO:teuthology.orchestra.run.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T12:35:19.819 INFO:teuthology.orchestra.run.smithi016.stdout: return _run_code(code, main_globals, None, 2024-09-17T12:35:19.819 INFO:teuthology.orchestra.run.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T12:35:19.820 INFO:teuthology.orchestra.run.smithi016.stdout: exec(code, run_globals) 2024-09-17T12:35:19.820 INFO:teuthology.orchestra.run.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5628, in 2024-09-17T12:35:19.820 INFO:teuthology.orchestra.run.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 5616, in main 2024-09-17T12:35:19.820 INFO:teuthology.orchestra.run.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-17T12:35:19.820 INFO:teuthology.orchestra.run.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-17T12:35:19.820 INFO:teuthology.orchestra.run.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-17T12:35:19.820 INFO:teuthology.orchestra.run.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 1111, in deploy_daemon 2024-09-17T12:35:19.820 INFO:teuthology.orchestra.run.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/__main__.py", line 1247, in deploy_daemon_units 2024-09-17T12:35:19.820 INFO:teuthology.orchestra.run.smithi016.stdout: File "/tmp/tmpw5qw9iga.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T12:35:19.820 INFO:teuthology.orchestra.run.smithi016.stdout:RuntimeError: Failed command: systemctl start ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca: Job for ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service failed because the control process exited with error code. 2024-09-17T12:35:19.820 INFO:teuthology.orchestra.run.smithi016.stdout:See "systemctl status ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" and "journalctl -xeu ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@iscsi.foo.smithi016.qaboca.service" for details. 2024-09-17T12:35:20.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:20 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/1813476612' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:35:20.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:20 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/1813476612' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:35:20.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:20 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/1813476612' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-17T12:35:20.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-09-17T12:35:20.767 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:35:21.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:21 smithi016 ceph-mon[87889]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:21.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:21 smithi016 ceph-mon[87889]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:35:21.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:21 smithi016 ceph-mon[89784]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:21.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:21 smithi016 ceph-mon[89784]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:35:21.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:21 smithi196 ceph-mon[80339]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:21.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:21 smithi196 ceph-mon[80339]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:35:22.664 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-09-17T12:35:23.218 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:23 smithi016 ceph-mon[87889]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:23.218 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:23 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/543414948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:35:23.218 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:23 smithi016 ceph-mon[89784]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:23.219 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:23 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/543414948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:35:23.238 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-09-17T12:35:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:35:23.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:23 smithi196 ceph-mon[80339]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:23.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:23 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/543414948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:35:25.125 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:25 smithi016 ceph-mon[87889]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:25.231 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.3.0-4937-g74984932 (7498493279982dfe87f00616198a5967475f6169) squid (dev)" 2024-09-17T12:35:25.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:25 smithi016 ceph-mon[89784]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:25.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:25 smithi196 ceph-mon[80339]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:25.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-09-17T12:35:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:35:26.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:26 smithi016 ceph-mon[87889]: from='client.? 172.21.15.16:0/4233294283' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:35:26.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:26 smithi016 ceph-mon[89784]: from='client.? 172.21.15.16:0/4233294283' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:35:26.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:26 smithi196 ceph-mon[80339]: from='client.? 172.21.15.16:0/4233294283' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-17T12:35:26.542 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:26 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:35:26] "GET /metrics HTTP/1.1" 200 38849 "" "Prometheus/2.43.0" 2024-09-17T12:35:27.439 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:27 smithi016 ceph-mon[87889]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:27.439 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:27 smithi016 ceph-mon[89784]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:27.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:27 smithi196 ceph-mon[80339]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:27.993 INFO:teuthology.orchestra.run.smithi016.stdout:osd 8 115s ago - 2024-09-17T12:35:28.668 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T12:35:28.679 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-09-17T12:35:28.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-09-17T12:35:28.888 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:35:29.266 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:29 smithi016 ceph-mon[89784]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:29.266 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:29 smithi016 ceph-mon[89784]: from='client.44774 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:29.266 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:29 smithi016 ceph-mon[87889]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:29.266 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:29 smithi016 ceph-mon[87889]: from='client.44774 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:29.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:29 smithi196 ceph-mon[80339]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:29.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:29 smithi196 ceph-mon[80339]: from='client.44774 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:30.975 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-17T12:35:30.975 INFO:teuthology.orchestra.run.smithi016.stdout: "image": "quay.io/ceph/ceph", 2024-09-17T12:35:30.975 INFO:teuthology.orchestra.run.smithi016.stdout: "registry": "quay.io", 2024-09-17T12:35:30.976 INFO:teuthology.orchestra.run.smithi016.stdout: "bare_image": "ceph/ceph", 2024-09-17T12:35:30.976 INFO:teuthology.orchestra.run.smithi016.stdout: "versions": [] 2024-09-17T12:35:30.976 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-17T12:35:31.302 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:31 smithi016 ceph-mon[87889]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:31.302 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:31 smithi016 ceph-mon[87889]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:35:31.302 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:31 smithi016 ceph-mon[89784]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:31.302 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:31 smithi016 ceph-mon[89784]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:35:31.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:31 smithi196 ceph-mon[80339]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:31.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:31 smithi196 ceph-mon[80339]: from='client.34722 -' entity='client.iscsi.foo.smithi016.qaboca' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T12:35:31.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-09-17T12:35:31.817 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:35:32.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:32 smithi016 ceph-mon[87889]: from='client.34839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:32.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:32 smithi016 ceph-mon[89784]: from='client.34839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:32.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:32 smithi196 ceph-mon[80339]: from='client.34839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:33.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:33 smithi016 ceph-mon[87889]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:33.441 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:33 smithi016 ceph-mon[89784]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:33.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:33 smithi196 ceph-mon[80339]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T12:35:33.835 INFO:teuthology.orchestra.run.smithi016.stdout: "16.2.0", 2024-09-17T12:35:34.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:35:34.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:34 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:35:34.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:34 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:35:34.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:34 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:35:34.431 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-09-17T12:35:34.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:34 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-17T12:35:34.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:34 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T12:35:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T12:35:35.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:35 smithi016 ceph-mon[87889]: from='client.44786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:35.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:35 smithi016 ceph-mon[87889]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:35.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:35 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:35:35.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:35 smithi016 ceph-mon[87889]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:35:35.441 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:35 smithi016 ceph-mon[87889]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:35:35.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:35 smithi016 ceph-mon[89784]: from='client.44786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:35.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:35 smithi016 ceph-mon[89784]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:35.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:35 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:35:35.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:35 smithi016 ceph-mon[89784]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:35:35.442 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:35 smithi016 ceph-mon[89784]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:35:35.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:35 smithi196 ceph-mon[80339]: from='client.44786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T12:35:35.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:35 smithi196 ceph-mon[80339]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T12:35:35.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:35 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T12:35:35.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:35 smithi196 ceph-mon[80339]: from='mgr.25283 172.21.15.16:0/3192858525' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T12:35:35.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:35 smithi196 ceph-mon[80339]: from='mgr.25283 ' entity='mgr.y' 2024-09-17T12:35:36.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:36 smithi016 ceph-mon[87889]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-09-17T12:35:36.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:36 smithi016 ceph-mon[87889]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:35:36.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:36 smithi016 ceph-mon[89784]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-09-17T12:35:36.192 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:36 smithi016 ceph-mon[89784]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:35:36.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:36 smithi196 ceph-mon[80339]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-09-17T12:35:36.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:36 smithi196 ceph-mon[80339]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T12:35:36.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:36 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: ::ffff:172.21.15.196 - - [17/Sep/2024:12:35:36] "GET /metrics HTTP/1.1" 200 38847 "" "Prometheus/2.43.0" 2024-09-17T12:35:36.717 INFO:teuthology.orchestra.run.smithi016.stdout: "v16.2.2", 2024-09-17T12:35:36.717 INFO:teuthology.orchestra.run.smithi016.stdout: "v16.2.2-20210505", 2024-09-17T12:35:37.301 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-17T12:35:37.311 INFO:tasks.cephadm:Teardown begin 2024-09-17T12:35:37.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T12:35:37.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T12:35:37.377 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-17T12:35:37.378 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T12:35:37.401 DEBUG:teuthology.orchestra.run.smithi196:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T12:35:37.435 INFO:tasks.cephadm:Stopping all daemons... 2024-09-17T12:35:37.435 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-17T12:35:37.436 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.a 2024-09-17T12:35:37.691 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:37 smithi016 systemd[1]: Stopping Ceph mon.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:35:38.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:37 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a[87885]: 2024-09-17T12:35:37.752+0000 7fb2f2a21640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:35:38.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:37 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a[87885]: 2024-09-17T12:35:37.752+0000 7fb2f2a21640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-17T12:35:38.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:37 smithi016 podman[125153]: 2024-09-17 12:35:37.783471676 +0000 UTC m=+0.148325953 container died 4a83b787c6059560782c44c0bbc408d98604596747a05d9ab1df11eeb2c274c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=main-7498493, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-09-17T12:35:38.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:38 smithi016 podman[125153]: 2024-09-17 12:35:38.185275376 +0000 UTC m=+0.550129652 container cleanup 4a83b787c6059560782c44c0bbc408d98604596747a05d9ab1df11eeb2c274c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a, RELEASE=main-7498493, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-17T12:35:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 17 12:35:38 smithi016 bash[125153]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-a 2024-09-17T12:35:38.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.a.service' 2024-09-17T12:35:38.619 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:35:38.619 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-17T12:35:38.620 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-17T12:35:38.620 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.c 2024-09-17T12:35:38.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:38 smithi016 systemd[1]: Stopping Ceph mon.c for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:35:39.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:35:38] ENGINE Bus STOPPING 2024-09-17T12:35:39.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c[89780]: 2024-09-17T12:35:38.915+0000 7f3f8a805640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:35:39.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:38 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c[89780]: 2024-09-17T12:35:38.915+0000 7f3f8a805640 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2024-09-17T12:35:39.191 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:39 smithi016 podman[125272]: 2024-09-17 12:35:39.088702723 +0000 UTC m=+0.280538675 container died aa5074acd22aae41d6cb06c18d239991acc17c796d6eda72a6813879f7af43f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-7498493, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-17T12:35:39.601 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:39 smithi016 podman[125272]: 2024-09-17 12:35:39.43221826 +0000 UTC m=+0.624054222 container cleanup aa5074acd22aae41d6cb06c18d239991acc17c796d6eda72a6813879f7af43f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=main-7498493, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-17T12:35:39.601 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:39 smithi016 bash[125272]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c 2024-09-17T12:35:39.601 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:35:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T12:35:39.602 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:35:39] ENGINE Bus STOPPED 2024-09-17T12:35:39.602 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:35:39] ENGINE Bus STARTING 2024-09-17T12:35:39.602 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:35:39] ENGINE Serving on http://:::9283 2024-09-17T12:35:39.602 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:39 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: [17/Sep/2024:12:35:39] ENGINE Bus STARTED 2024-09-17T12:35:39.852 INFO:journalctl@ceph.mon.c.smithi016.stdout:Sep 17 12:35:39 smithi016 podman[125284]: 2024-09-17 12:35:39.600142496 +0000 UTC m=+0.505548703 container remove aa5074acd22aae41d6cb06c18d239991acc17c796d6eda72a6813879f7af43f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=main-7498493, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-17T12:35:39.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.c.service' 2024-09-17T12:35:39.892 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:35:39.893 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-17T12:35:39.893 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-17T12:35:39.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.b 2024-09-17T12:35:40.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:39 smithi196 systemd[1]: Stopping Ceph mon.b for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:35:40.459 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:40 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b[80335]: 2024-09-17T12:35:40.203+0000 7f3460bf3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:35:40.459 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:40 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b[80335]: 2024-09-17T12:35:40.203+0000 7f3460bf3640 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2024-09-17T12:35:40.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:40 smithi196 podman[108856]: 2024-09-17 12:35:40.458336877 +0000 UTC m=+0.365373219 container died dc48dfc9e40cb61472b185c889f77662327130265661fdd9ef1d75eaf04ceb5a (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-7498493, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True) 2024-09-17T12:35:41.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:40 smithi196 podman[108856]: 2024-09-17 12:35:40.886729806 +0000 UTC m=+0.793766146 container cleanup dc48dfc9e40cb61472b185c889f77662327130265661fdd9ef1d75eaf04ceb5a (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-7498493, GIT_CLEAN=True) 2024-09-17T12:35:41.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:40 smithi196 bash[108856]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b 2024-09-17T12:35:41.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 17 12:35:41 smithi196 podman[108869]: 2024-09-17 12:35:41.105007147 +0000 UTC m=+0.643223365 container remove dc48dfc9e40cb61472b185c889f77662327130265661fdd9ef1d75eaf04ceb5a (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mon-b, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=main-7498493, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:35:41.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mon.b.service' 2024-09-17T12:35:41.412 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:35:41.413 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-17T12:35:41.413 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-17T12:35:41.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.y 2024-09-17T12:35:41.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:41 smithi016 systemd[1]: Stopping Ceph mgr.y for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:35:42.063 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:41 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y[79092]: 2024-09-17T12:35:41.728+0000 7f9d95f0e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:35:42.063 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:41 smithi016 podman[125404]: 2024-09-17 12:35:41.759141495 +0000 UTC m=+0.140266201 container died 953d9c4b1666ffecc5bd796e5149e6bfb8e239107f773f99dfe564c29c44a8e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-17T12:35:42.397 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:42 smithi016 podman[125404]: 2024-09-17 12:35:42.144919831 +0000 UTC m=+0.526044529 container cleanup 953d9c4b1666ffecc5bd796e5149e6bfb8e239107f773f99dfe564c29c44a8e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=main-7498493, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-17T12:35:42.397 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Sep 17 12:35:42 smithi016 bash[125404]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-y 2024-09-17T12:35:42.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-17T12:35:42.613 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:35:42.613 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-17T12:35:42.613 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-17T12:35:42.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.x 2024-09-17T12:35:43.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:35:42 smithi196 systemd[1]: Stopping Ceph mgr.x for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:35:43.526 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:35:43 smithi196 podman[108980]: 2024-09-17 12:35:43.213889023 +0000 UTC m=+0.405465989 container died 89153031794c53c564f534571bfcdf43d71f84cd952570bb2e7f709442c328c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-7498493, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-17T12:35:43.851 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:35:43 smithi196 podman[108980]: 2024-09-17 12:35:43.624205355 +0000 UTC m=+0.815782323 container cleanup 89153031794c53c564f534571bfcdf43d71f84cd952570bb2e7f709442c328c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-7498493, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-09-17T12:35:43.852 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 17 12:35:43 smithi196 bash[108980]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-mgr-x 2024-09-17T12:35:43.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-17T12:35:44.034 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:35:44.034 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-17T12:35:44.034 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-17T12:35:44.034 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.0 2024-09-17T12:35:44.325 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:44 smithi016 systemd[1]: Stopping Ceph osd.0 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:35:44.691 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:44 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0[94947]: 2024-09-17T12:35:44.323+0000 7fdc02a81640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:35:44.691 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:44 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0[94947]: 2024-09-17T12:35:44.323+0000 7fdc02a81640 -1 osd.0 151 *** Got signal Terminated *** 2024-09-17T12:35:44.691 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:44 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0[94947]: 2024-09-17T12:35:44.323+0000 7fdc02a81640 -1 osd.0 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:35:50.192 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:49 smithi016 podman[125525]: 2024-09-17 12:35:49.859630679 +0000 UTC m=+5.629253022 container died d12e6f26f048fa40ae36e598638f61fb01c68ec4d50c9ad7832d93c89e277c61 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-7498493, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T12:35:50.493 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:50 smithi016 podman[125525]: 2024-09-17 12:35:50.210306111 +0000 UTC m=+5.979928460 container cleanup d12e6f26f048fa40ae36e598638f61fb01c68ec4d50c9ad7832d93c89e277c61 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-7498493, org.label-schema.build-date=20240731) 2024-09-17T12:35:50.493 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:50 smithi016 bash[125525]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0 2024-09-17T12:35:50.844 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:50 smithi016 podman[125540]: 2024-09-17 12:35:50.492437367 +0000 UTC m=+0.627761131 container remove d12e6f26f048fa40ae36e598638f61fb01c68ec4d50c9ad7832d93c89e277c61 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-7498493, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True) 2024-09-17T12:35:51.097 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:50 smithi016 podman[125631]: 2024-09-17 12:35:50.843392196 +0000 UTC m=+0.112521397 container create c57678658155c9c251bcd4715baa1e7bef01c40158db74540cd98e48c973434b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-7498493, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-17T12:35:51.097 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:50 smithi016 podman[125631]: 2024-09-17 12:35:50.74784054 +0000 UTC m=+0.016969743 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:35:51.097 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:51 smithi016 podman[125631]: 2024-09-17 12:35:51.037743294 +0000 UTC m=+0.306872492 container init c57678658155c9c251bcd4715baa1e7bef01c40158db74540cd98e48c973434b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-7498493, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-17T12:35:51.097 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:51 smithi016 podman[125631]: 2024-09-17 12:35:51.041092855 +0000 UTC m=+0.310222049 container start c57678658155c9c251bcd4715baa1e7bef01c40158db74540cd98e48c973434b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=main-7498493, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-17T12:35:51.097 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:51 smithi016 podman[125631]: 2024-09-17 12:35:51.095889958 +0000 UTC m=+0.365019168 container attach c57678658155c9c251bcd4715baa1e7bef01c40158db74540cd98e48c973434b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=main-7498493, org.label-schema.schema-version=1.0) 2024-09-17T12:35:51.441 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:51 smithi016 podman[125631]: 2024-09-17 12:35:51.231003047 +0000 UTC m=+0.500132241 container died c57678658155c9c251bcd4715baa1e7bef01c40158db74540cd98e48c973434b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-17T12:35:51.811 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 17 12:35:51 smithi016 podman[125654]: 2024-09-17 12:35:51.783058604 +0000 UTC m=+0.545776918 container remove c57678658155c9c251bcd4715baa1e7bef01c40158db74540cd98e48c973434b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-7498493, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:35:51.817 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.0.service' 2024-09-17T12:35:51.860 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:35:51.860 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-17T12:35:51.860 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-17T12:35:51.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.1 2024-09-17T12:35:52.338 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:52 smithi016 systemd[1]: Stopping Ceph osd.1 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:35:52.691 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:52 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1[99692]: 2024-09-17T12:35:52.335+0000 7f5fbd321640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:35:52.691 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:52 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1[99692]: 2024-09-17T12:35:52.335+0000 7f5fbd321640 -1 osd.1 151 *** Got signal Terminated *** 2024-09-17T12:35:52.691 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:52 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1[99692]: 2024-09-17T12:35:52.335+0000 7f5fbd321640 -1 osd.1 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:35:54.442 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:35:54 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:35:54.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-09-17T12:35:54.442 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:35:54 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:35:54.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-17T12:35:58.080 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:57 smithi016 podman[125737]: 2024-09-17 12:35:57.812863999 +0000 UTC m=+5.590967190 container died cd2f6e2964dd8a6489d4313a1176ed6e69f77cc1871bfd4eafe63fdc4f0d224b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-7498493, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-17T12:35:58.380 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:58 smithi016 podman[125737]: 2024-09-17 12:35:58.136378883 +0000 UTC m=+5.914482065 container cleanup cd2f6e2964dd8a6489d4313a1176ed6e69f77cc1871bfd4eafe63fdc4f0d224b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=main-7498493, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-17T12:35:58.380 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:58 smithi016 bash[125737]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1 2024-09-17T12:35:58.691 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:58 smithi016 podman[125766]: 2024-09-17 12:35:58.379262489 +0000 UTC m=+0.558735157 container remove cd2f6e2964dd8a6489d4313a1176ed6e69f77cc1871bfd4eafe63fdc4f0d224b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1, RELEASE=main-7498493, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-17T12:35:59.125 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:58 smithi016 podman[125858]: 2024-09-17 12:35:58.718444993 +0000 UTC m=+0.016485113 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:35:59.126 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:58 smithi016 podman[125858]: 2024-09-17 12:35:58.847146088 +0000 UTC m=+0.145186179 container create 40a2c2c30a2fca66b16eeee2b0dfba0d93e7b6fd1f55162a67b3c777bd074f3d (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=main-7498493, GIT_BRANCH=HEAD) 2024-09-17T12:35:59.126 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:59 smithi016 podman[125858]: 2024-09-17 12:35:59.049536765 +0000 UTC m=+0.347576880 container init 40a2c2c30a2fca66b16eeee2b0dfba0d93e7b6fd1f55162a67b3c777bd074f3d (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=main-7498493, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-17T12:35:59.126 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:59 smithi016 podman[125858]: 2024-09-17 12:35:59.052986401 +0000 UTC m=+0.351026494 container start 40a2c2c30a2fca66b16eeee2b0dfba0d93e7b6fd1f55162a67b3c777bd074f3d (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-7498493, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-17T12:35:59.126 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:59 smithi016 podman[125858]: 2024-09-17 12:35:59.124500789 +0000 UTC m=+0.422540892 container attach 40a2c2c30a2fca66b16eeee2b0dfba0d93e7b6fd1f55162a67b3c777bd074f3d (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, RELEASE=main-7498493, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:35:59.441 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:59 smithi016 podman[125858]: 2024-09-17 12:35:59.248938411 +0000 UTC m=+0.546978525 container died 40a2c2c30a2fca66b16eeee2b0dfba0d93e7b6fd1f55162a67b3c777bd074f3d (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-7498493, org.label-schema.build-date=20240731) 2024-09-17T12:35:59.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.1.service' 2024-09-17T12:35:59.797 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:59 smithi016 podman[125882]: 2024-09-17 12:35:59.744870021 +0000 UTC m=+0.489822792 container remove 40a2c2c30a2fca66b16eeee2b0dfba0d93e7b6fd1f55162a67b3c777bd074f3d (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-7498493, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-17T12:35:59.797 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:59 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.1.service: Deactivated successfully. 2024-09-17T12:35:59.797 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:59 smithi016 systemd[1]: Stopped Ceph osd.1 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:35:59.797 INFO:journalctl@ceph.osd.1.smithi016.stdout:Sep 17 12:35:59 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.1.service: Consumed 6.935s CPU time. 2024-09-17T12:35:59.814 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:35:59.815 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-17T12:35:59.815 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-17T12:35:59.815 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.2 2024-09-17T12:36:00.166 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:35:59 smithi016 systemd[1]: Stopping Ceph osd.2 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:36:00.441 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:36:00 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2[104705]: 2024-09-17T12:36:00.164+0000 7f80a4c97640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:36:00.441 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:36:00 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2[104705]: 2024-09-17T12:36:00.164+0000 7f80a4c97640 -1 osd.2 151 *** Got signal Terminated *** 2024-09-17T12:36:00.441 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:36:00 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2[104705]: 2024-09-17T12:36:00.164+0000 7f80a4c97640 -1 osd.2 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:36:04.691 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:04 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:04.241Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-09-17T12:36:04.691 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:04 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:04.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-09-17T12:36:04.692 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:04 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:04.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-17T12:36:05.793 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:36:05 smithi016 podman[125962]: 2024-09-17 12:36:05.505916854 +0000 UTC m=+5.464495355 container died 99430f24b824e85c79ab33304fb9373a3eaef586c57aadccf4d8b95834dc0865 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-7498493, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-17T12:36:06.125 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:36:05 smithi016 podman[125962]: 2024-09-17 12:36:05.864868433 +0000 UTC m=+5.823446937 container cleanup 99430f24b824e85c79ab33304fb9373a3eaef586c57aadccf4d8b95834dc0865 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-7498493, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-17T12:36:06.125 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:36:05 smithi016 bash[125962]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2 2024-09-17T12:36:06.441 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:36:06 smithi016 podman[125980]: 2024-09-17 12:36:06.124240347 +0000 UTC m=+0.612445558 container remove 99430f24b824e85c79ab33304fb9373a3eaef586c57aadccf4d8b95834dc0865 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-7498493, CEPH_POINT_RELEASE=) 2024-09-17T12:36:06.787 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:36:06 smithi016 podman[126068]: 2024-09-17 12:36:06.475230504 +0000 UTC m=+0.101353646 container create f70b56f00e5491b630ac3065f88a6f40de91645abde1fd02d0effd2e1834bdcf (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-7498493, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-17T12:36:06.787 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:36:06 smithi016 podman[126068]: 2024-09-17 12:36:06.390399784 +0000 UTC m=+0.016522925 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:36:06.787 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:36:06 smithi016 podman[126068]: 2024-09-17 12:36:06.694445095 +0000 UTC m=+0.320568242 container init f70b56f00e5491b630ac3065f88a6f40de91645abde1fd02d0effd2e1834bdcf (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=main-7498493) 2024-09-17T12:36:06.787 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:36:06 smithi016 podman[126068]: 2024-09-17 12:36:06.697921277 +0000 UTC m=+0.324044427 container start f70b56f00e5491b630ac3065f88a6f40de91645abde1fd02d0effd2e1834bdcf (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=main-7498493, GIT_BRANCH=HEAD) 2024-09-17T12:36:06.787 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:36:06 smithi016 podman[126068]: 2024-09-17 12:36:06.785982724 +0000 UTC m=+0.412105878 container attach f70b56f00e5491b630ac3065f88a6f40de91645abde1fd02d0effd2e1834bdcf (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=main-7498493, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-17T12:36:07.190 INFO:journalctl@ceph.osd.2.smithi016.stdout:Sep 17 12:36:06 smithi016 podman[126068]: 2024-09-17 12:36:06.893545907 +0000 UTC m=+0.519669049 container died f70b56f00e5491b630ac3065f88a6f40de91645abde1fd02d0effd2e1834bdcf (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-7498493, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-17T12:36:07.468 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.2.service' 2024-09-17T12:36:07.511 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:36:07.511 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-17T12:36:07.511 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-17T12:36:07.512 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.3 2024-09-17T12:36:07.844 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:07 smithi016 systemd[1]: Stopping Ceph osd.3 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:36:08.191 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:07 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3[109554]: 2024-09-17T12:36:07.842+0000 7fa322a62640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:36:08.191 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:07 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3[109554]: 2024-09-17T12:36:07.842+0000 7fa322a62640 -1 osd.3 151 *** Got signal Terminated *** 2024-09-17T12:36:08.191 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:07 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3[109554]: 2024-09-17T12:36:07.842+0000 7fa322a62640 -1 osd.3 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:36:09.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:08.883Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-09-17T12:36:09.280 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:08.883Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-09-17T12:36:09.280 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:08.888Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-09-17T12:36:09.280 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:08.888Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-09-17T12:36:09.280 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:08.888Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-09-17T12:36:09.280 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:08.889Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-09-17T12:36:09.280 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:08 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:08.889Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-09-17T12:36:11.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:11 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:11.110+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:12.191 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:11 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3[109554]: 2024-09-17T12:36:11.705+0000 7fa31f077640 -1 osd.3 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.577258+0000 front 2024-09-17T12:35:47.577362+0000 (oldest deadline 2024-09-17T12:36:11.077237+0000) 2024-09-17T12:36:12.471 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:12 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:12.133+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:12.776 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:12 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:12.469+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:13.191 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:12 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3[109554]: 2024-09-17T12:36:12.713+0000 7fa31f077640 -1 osd.3 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.577258+0000 front 2024-09-17T12:35:47.577362+0000 (oldest deadline 2024-09-17T12:36:11.077237+0000) 2024-09-17T12:36:13.276 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:13 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:36:13.047+0000 7f164da8f640 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.421601+0000 front 2024-09-17T12:35:47.421521+0000 (oldest deadline 2024-09-17T12:36:12.721406+0000) 2024-09-17T12:36:13.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:13 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:13.002+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:13.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:13 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:13.126+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:13.535 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:13 smithi016 podman[126171]: 2024-09-17 12:36:13.238615691 +0000 UTC m=+5.491214834 container died 1d4309778a0d2596b6d7276f16e373eb1dfa17a1e93b97e23ce883c74d609e24 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=main-7498493, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-17T12:36:13.535 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:13 smithi016 podman[126171]: 2024-09-17 12:36:13.53397872 +0000 UTC m=+5.786577865 container cleanup 1d4309778a0d2596b6d7276f16e373eb1dfa17a1e93b97e23ce883c74d609e24 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=main-7498493, ceph=True, GIT_CLEAN=True) 2024-09-17T12:36:13.776 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:13 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:13.475+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:13.941 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:13 smithi016 bash[126171]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3 2024-09-17T12:36:13.941 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:13 smithi016 podman[126186]: 2024-09-17 12:36:13.768622019 +0000 UTC m=+0.522372946 container remove 1d4309778a0d2596b6d7276f16e373eb1dfa17a1e93b97e23ce883c74d609e24 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=main-7498493, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-17T12:36:14.276 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:14 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:36:14.013+0000 7f164da8f640 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.421601+0000 front 2024-09-17T12:35:47.421521+0000 (oldest deadline 2024-09-17T12:36:12.721406+0000) 2024-09-17T12:36:14.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:14 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:14.009+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:14.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:14 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:14.136+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:14.348 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:14 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:14.242Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-17T12:36:14.348 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:14 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:14.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-09-17T12:36:14.348 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:14 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:14.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-17T12:36:14.348 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:14 smithi016 podman[126276]: 2024-09-17 12:36:14.086182839 +0000 UTC m=+0.109860998 container create 45413d05cd3a174b3e1ff8125eb72c2dc82938a7ff86f838af22ec42f309a743 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-7498493, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-17T12:36:14.348 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:14 smithi016 podman[126276]: 2024-09-17 12:36:13.992915144 +0000 UTC m=+0.016593270 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:36:14.349 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:14 smithi016 podman[126276]: 2024-09-17 12:36:14.2721557 +0000 UTC m=+0.295833829 container init 45413d05cd3a174b3e1ff8125eb72c2dc82938a7ff86f838af22ec42f309a743 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=main-7498493, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T12:36:14.349 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:14 smithi016 podman[126276]: 2024-09-17 12:36:14.275668191 +0000 UTC m=+0.299346307 container start 45413d05cd3a174b3e1ff8125eb72c2dc82938a7ff86f838af22ec42f309a743 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-7498493, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-17T12:36:14.349 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:14 smithi016 podman[126276]: 2024-09-17 12:36:14.347066375 +0000 UTC m=+0.370744495 container attach 45413d05cd3a174b3e1ff8125eb72c2dc82938a7ff86f838af22ec42f309a743 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-7498493, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-17T12:36:14.691 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:14 smithi016 podman[126276]: 2024-09-17 12:36:14.472848508 +0000 UTC m=+0.496526632 container died 45413d05cd3a174b3e1ff8125eb72c2dc82938a7ff86f838af22ec42f309a743 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=main-7498493, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-17T12:36:14.776 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:14 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:14.441+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:15.057 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 17 12:36:15 smithi016 podman[126299]: 2024-09-17 12:36:15.04228115 +0000 UTC m=+0.563364570 container remove 45413d05cd3a174b3e1ff8125eb72c2dc82938a7ff86f838af22ec42f309a743 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-09-17T12:36:15.063 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.3.service' 2024-09-17T12:36:15.105 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:36:15.106 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-17T12:36:15.106 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-17T12:36:15.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.4 2024-09-17T12:36:15.276 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:15.051+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:15.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:15.181+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:15.277 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:36:15.016+0000 7f164da8f640 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.421601+0000 front 2024-09-17T12:35:47.421521+0000 (oldest deadline 2024-09-17T12:36:12.721406+0000) 2024-09-17T12:36:15.277 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:15 smithi196 systemd[1]: Stopping Ceph osd.4 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:36:15.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:15.445+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:15.777 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:36:15.416+0000 7f165147a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:36:15.777 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:36:15.416+0000 7f165147a640 -1 osd.4 151 *** Got signal Terminated *** 2024-09-17T12:36:15.777 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:15 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:36:15.416+0000 7f165147a640 -1 osd.4 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:36:16.276 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:36:16.000+0000 7f164da8f640 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.421601+0000 front 2024-09-17T12:35:47.421521+0000 (oldest deadline 2024-09-17T12:36:12.721406+0000) 2024-09-17T12:36:16.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:16.097+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:16.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:16.206+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:16.776 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:16.449+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:17.277 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:16 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:36:16.972+0000 7f164da8f640 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.421601+0000 front 2024-09-17T12:35:47.421521+0000 (oldest deadline 2024-09-17T12:36:12.721406+0000) 2024-09-17T12:36:17.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:17.058+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:17.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:17.189+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:17.776 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:17.452+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:18.165 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:36:17.945+0000 7f164da8f640 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.421601+0000 front 2024-09-17T12:35:47.421521+0000 (oldest deadline 2024-09-17T12:36:12.721406+0000) 2024-09-17T12:36:18.166 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:18.037+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:18.166 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:36:17 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a[56313]: logger=sqlstore.transactions t=2024-09-17T12:36:17.852950634Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T12:36:18.464 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:18.164+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:18.776 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:18.462+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:18.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:18.462+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:19.276 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:18 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:36:18.958+0000 7f164da8f640 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.421601+0000 front 2024-09-17T12:35:47.421521+0000 (oldest deadline 2024-09-17T12:36:12.721406+0000) 2024-09-17T12:36:19.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:19 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:19.080+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:19.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:19 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:19.147+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:19.776 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:19 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:19.480+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:19.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:19 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:19.480+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:20.276 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:19 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4[85222]: 2024-09-17T12:36:19.949+0000 7f164da8f640 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.421601+0000 front 2024-09-17T12:35:47.421521+0000 (oldest deadline 2024-09-17T12:36:12.721406+0000) 2024-09-17T12:36:20.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:20 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:20.127+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:20.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:20 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:20.133+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:20.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:20 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:20.496+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:20.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:20 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:20.496+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:21.235 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:20 smithi196 podman[109116]: 2024-09-17 12:36:20.948379315 +0000 UTC m=+5.641878123 container died 0c739e0cd187a4a6e87714eaa5ab8ddefdc5669f29997a2b89a66fafb2ea866c (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-7498493) 2024-09-17T12:36:21.236 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:21 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:21.114+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:21.236 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:21 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:21.114+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.632022+0000 front 2024-09-17T12:35:56.632134+0000 (oldest deadline 2024-09-17T12:36:20.131913+0000) 2024-09-17T12:36:21.236 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:21 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:21.152+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:21.236 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:21 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:21.152+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:21.526 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:21 smithi196 podman[109116]: 2024-09-17 12:36:21.408716012 +0000 UTC m=+6.102214858 container cleanup 0c739e0cd187a4a6e87714eaa5ab8ddefdc5669f29997a2b89a66fafb2ea866c (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, RELEASE=main-7498493, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-17T12:36:21.526 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:21 smithi196 bash[109116]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4 2024-09-17T12:36:22.022 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:21 smithi196 podman[109132]: 2024-09-17 12:36:21.701635273 +0000 UTC m=+0.742988747 container remove 0c739e0cd187a4a6e87714eaa5ab8ddefdc5669f29997a2b89a66fafb2ea866c (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=main-7498493, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-17T12:36:22.022 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:21 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:21.538+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:22.022 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:21 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:21.538+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:22.276 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:22 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:22.122+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:22.277 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:22 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:22.122+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.632022+0000 front 2024-09-17T12:35:56.632134+0000 (oldest deadline 2024-09-17T12:36:20.131913+0000) 2024-09-17T12:36:22.277 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:22 smithi196 podman[109220]: 2024-09-17 12:36:21.923498691 +0000 UTC m=+0.015916208 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:36:22.277 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:22 smithi196 podman[109220]: 2024-09-17 12:36:22.052616897 +0000 UTC m=+0.145034459 container create 270125e59f594620c0da498592a7cfc4fe21e70ca62dded84bfe6136130cc03b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=main-7498493, ceph=True) 2024-09-17T12:36:22.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:22 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:22.143+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:22.278 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:22 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:22.143+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:22.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:22 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:22.524+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:22.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:22 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:22.524+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:22.777 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:22 smithi196 podman[109220]: 2024-09-17 12:36:22.280064835 +0000 UTC m=+0.372482360 container init 270125e59f594620c0da498592a7cfc4fe21e70ca62dded84bfe6136130cc03b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-7498493, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T12:36:22.777 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:22 smithi196 podman[109220]: 2024-09-17 12:36:22.283564417 +0000 UTC m=+0.375981937 container start 270125e59f594620c0da498592a7cfc4fe21e70ca62dded84bfe6136130cc03b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-7498493, org.label-schema.schema-version=1.0) 2024-09-17T12:36:22.777 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:22 smithi196 podman[109220]: 2024-09-17 12:36:22.371714062 +0000 UTC m=+0.464131591 container attach 270125e59f594620c0da498592a7cfc4fe21e70ca62dded84bfe6136130cc03b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-7498493, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True) 2024-09-17T12:36:22.777 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 17 12:36:22 smithi196 podman[109220]: 2024-09-17 12:36:22.483215354 +0000 UTC m=+0.575632874 container died 270125e59f594620c0da498592a7cfc4fe21e70ca62dded84bfe6136130cc03b (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-7498493, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-17T12:36:23.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.4.service' 2024-09-17T12:36:23.131 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:36:23.132 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-17T12:36:23.132 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-17T12:36:23.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.5 2024-09-17T12:36:23.462 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:23 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:23.142+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:23.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:23 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:23.142+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.632022+0000 front 2024-09-17T12:35:56.632134+0000 (oldest deadline 2024-09-17T12:36:20.131913+0000) 2024-09-17T12:36:23.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:23 smithi196 systemd[1]: Stopping Ceph osd.5 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:36:23.463 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:23 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:23.139+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:23.463 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:23 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:23.139+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:23.776 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:23 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:23.460+0000 7fb321de1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:36:23.776 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:23 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:23.460+0000 7fb321de1640 -1 osd.5 151 *** Got signal Terminated *** 2024-09-17T12:36:23.777 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:23 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:23.460+0000 7fb321de1640 -1 osd.5 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:36:23.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:23 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:23.489+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:23.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:23 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:23.489+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:24.452 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:24 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:24.153+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:24.452 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:24 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:24.153+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:24.453 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:24 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:24.154+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:24.453 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:24 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:24.154+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.632022+0000 front 2024-09-17T12:35:56.632134+0000 (oldest deadline 2024-09-17T12:36:20.131913+0000) 2024-09-17T12:36:24.691 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:24 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:24.243Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-09-17T12:36:24.691 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:24 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:24.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-09-17T12:36:24.691 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:24 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:24.244Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-17T12:36:24.776 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:24 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:24.450+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:24.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:24 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:24.450+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:25.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:25 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:25.186+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:25.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:25 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:25.186+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:25.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:25 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:25.419+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:25.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:25 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:25.419+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:25.527 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:25 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:25.194+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:25.527 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:25 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:25.194+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.632022+0000 front 2024-09-17T12:35:56.632134+0000 (oldest deadline 2024-09-17T12:36:20.131913+0000) 2024-09-17T12:36:26.527 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:26 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:26.242+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:26.527 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:26 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:26.242+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.632022+0000 front 2024-09-17T12:35:56.632134+0000 (oldest deadline 2024-09-17T12:36:20.131913+0000) 2024-09-17T12:36:26.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:26 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:26.233+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:26.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:26 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:26.233+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:26.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:26 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:26.375+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:26.528 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:26 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:26.375+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:27.526 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:27 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:27.251+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:27.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:27 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:27.251+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:27.527 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:27 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:27.231+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:27.527 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:27 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:27.231+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.632022+0000 front 2024-09-17T12:35:56.632134+0000 (oldest deadline 2024-09-17T12:36:20.131913+0000) 2024-09-17T12:36:27.527 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:27 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:27.231+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:04.733253+0000 front 2024-09-17T12:36:04.733214+0000 (oldest deadline 2024-09-17T12:36:27.033080+0000) 2024-09-17T12:36:27.528 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:27 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:27.330+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:27.528 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:27 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:27.330+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:27.528 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:27 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:27.330+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:28.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:28 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:28.274+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:28.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:28 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:28.274+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:28.527 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:28 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:28.183+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.130833+0000 front 2024-09-17T12:35:49.130719+0000 (oldest deadline 2024-09-17T12:36:12.630607+0000) 2024-09-17T12:36:28.527 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:28 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:28.183+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.632022+0000 front 2024-09-17T12:35:56.632134+0000 (oldest deadline 2024-09-17T12:36:20.131913+0000) 2024-09-17T12:36:28.527 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:28 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5[89546]: 2024-09-17T12:36:28.183+0000 7fb31e3f6640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:04.733253+0000 front 2024-09-17T12:36:04.733214+0000 (oldest deadline 2024-09-17T12:36:27.033080+0000) 2024-09-17T12:36:28.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:28 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:28.375+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:28.528 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:28 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:28.375+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:28.528 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:28 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:28.375+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:29.079 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:28 smithi196 podman[109323]: 2024-09-17 12:36:28.807502938 +0000 UTC m=+5.450622959 container died 71f93c731227c5eec5e895f24bde851f003f3a358eaa75e9892987fe597964af (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-7498493, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:36:29.411 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:29 smithi196 podman[109323]: 2024-09-17 12:36:29.277578312 +0000 UTC m=+5.920698325 container cleanup 71f93c731227c5eec5e895f24bde851f003f3a358eaa75e9892987fe597964af (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=main-7498493, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-17T12:36:29.412 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:29 smithi196 bash[109323]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5 2024-09-17T12:36:29.412 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:29 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:29.225+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:29.412 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:29 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:29.225+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:29.412 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:29 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:29.225+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:03.156330+0000 front 2024-09-17T12:36:03.156386+0000 (oldest deadline 2024-09-17T12:36:28.456110+0000) 2024-09-17T12:36:29.776 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:29 smithi196 podman[109338]: 2024-09-17 12:36:29.537587465 +0000 UTC m=+0.731213998 container remove 71f93c731227c5eec5e895f24bde851f003f3a358eaa75e9892987fe597964af (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-7498493) 2024-09-17T12:36:29.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:29 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:29.409+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:29.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:29 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:29.409+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:29.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:29 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:29.409+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:30.150 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:29 smithi196 podman[109425]: 2024-09-17 12:36:29.797811111 +0000 UTC m=+0.016162925 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:36:30.150 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:29 smithi196 podman[109425]: 2024-09-17 12:36:29.904879014 +0000 UTC m=+0.123230807 container create ac2b3cfd86ca6f52a717b97254b4cc084e2beb57e89e3f24cfb548a76f2bf3d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=main-7498493, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-17T12:36:30.150 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:30 smithi196 podman[109425]: 2024-09-17 12:36:30.149082577 +0000 UTC m=+0.367434391 container init ac2b3cfd86ca6f52a717b97254b4cc084e2beb57e89e3f24cfb548a76f2bf3d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, RELEASE=main-7498493, GIT_CLEAN=True) 2024-09-17T12:36:30.431 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:30 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:30.184+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:30.431 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:30 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:30.184+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:30.431 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:30 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:30.184+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:03.156330+0000 front 2024-09-17T12:36:03.156386+0000 (oldest deadline 2024-09-17T12:36:28.456110+0000) 2024-09-17T12:36:30.431 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:30 smithi196 podman[109425]: 2024-09-17 12:36:30.152361664 +0000 UTC m=+0.370713468 container start ac2b3cfd86ca6f52a717b97254b4cc084e2beb57e89e3f24cfb548a76f2bf3d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=main-7498493, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-17T12:36:30.431 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:30 smithi196 podman[109425]: 2024-09-17 12:36:30.240718205 +0000 UTC m=+0.459070013 container attach ac2b3cfd86ca6f52a717b97254b4cc084e2beb57e89e3f24cfb548a76f2bf3d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-7498493, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-17T12:36:30.432 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 17 12:36:30 smithi196 podman[109425]: 2024-09-17 12:36:30.344262305 +0000 UTC m=+0.562614109 container died ac2b3cfd86ca6f52a717b97254b4cc084e2beb57e89e3f24cfb548a76f2bf3d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-7498493, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-17T12:36:30.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:30 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:30.429+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:30.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:30 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:30.429+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:30.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:30 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:30.429+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:30.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.5.service' 2024-09-17T12:36:30.960 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:36:30.960 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-17T12:36:30.960 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-17T12:36:30.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.6 2024-09-17T12:36:31.175 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:31 smithi196 systemd[1]: Stopping Ceph osd.6 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:36:31.453 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:31.173+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:31.453 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:31.173+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:31.453 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:31.173+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:03.156330+0000 front 2024-09-17T12:36:03.156386+0000 (oldest deadline 2024-09-17T12:36:28.456110+0000) 2024-09-17T12:36:31.453 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:31.288+0000 7f8d5a8fe640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:36:31.453 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:31.288+0000 7f8d5a8fe640 -1 osd.6 151 *** Got signal Terminated *** 2024-09-17T12:36:31.454 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:31.288+0000 7f8d5a8fe640 -1 osd.6 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:36:31.776 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:31.450+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:31.776 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:31.450+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:31.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:31 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:31.450+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:32.526 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:32 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:32.219+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:32.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:32 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:32.219+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:32.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:32 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:32.219+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:03.156330+0000 front 2024-09-17T12:36:03.156386+0000 (oldest deadline 2024-09-17T12:36:28.456110+0000) 2024-09-17T12:36:32.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:32 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:32.449+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:32.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:32 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:32.449+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:32.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:32 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:32.449+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:33.526 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:33.259+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:33.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:33.259+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:33.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:33.259+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:03.156330+0000 front 2024-09-17T12:36:03.156386+0000 (oldest deadline 2024-09-17T12:36:28.456110+0000) 2024-09-17T12:36:33.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:33.449+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:33.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:33.449+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:33.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:33 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:33.449+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:34.691 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:34 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:34.244Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-17T12:36:34.692 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:34 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:34.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-09-17T12:36:34.692 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:34 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:34.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-17T12:36:34.777 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:34.301+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:34.777 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:34.301+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:34.777 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:34.301+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:03.156330+0000 front 2024-09-17T12:36:03.156386+0000 (oldest deadline 2024-09-17T12:36:28.456110+0000) 2024-09-17T12:36:34.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:34.440+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:34.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:34.440+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:34.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:34 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:34.440+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:35.777 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:35.327+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:47.053104+0000 front 2024-09-17T12:35:47.053147+0000 (oldest deadline 2024-09-17T12:36:10.553045+0000) 2024-09-17T12:36:35.777 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:35.327+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.854962+0000 front 2024-09-17T12:35:56.855057+0000 (oldest deadline 2024-09-17T12:36:20.954985+0000) 2024-09-17T12:36:35.777 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6[93911]: 2024-09-17T12:36:35.327+0000 7f8d56f13640 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:03.156330+0000 front 2024-09-17T12:36:03.156386+0000 (oldest deadline 2024-09-17T12:36:28.456110+0000) 2024-09-17T12:36:35.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:35.411+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:35.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:35.411+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:35.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:35.411+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:35.778 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:35 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:35.411+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-09-17T12:36:09.923848+0000 front 2024-09-17T12:36:09.923907+0000 (oldest deadline 2024-09-17T12:36:34.623808+0000) 2024-09-17T12:36:36.720 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:36.422+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:36.720 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:36.422+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:36.720 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:36.422+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:36.720 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:36 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:36.422+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-09-17T12:36:09.923848+0000 front 2024-09-17T12:36:09.923907+0000 (oldest deadline 2024-09-17T12:36:34.623808+0000) 2024-09-17T12:36:36.990 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:36 smithi196 podman[109527]: 2024-09-17 12:36:36.718949321 +0000 UTC m=+5.531456924 container died 26b8884ca832ed59d4dbb7e1754805b9b59dbf8e12f57bc5961f51391c7953f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-7498493, CEPH_POINT_RELEASE=) 2024-09-17T12:36:37.276 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:37 smithi196 podman[109527]: 2024-09-17 12:36:37.163191069 +0000 UTC m=+5.975698716 container cleanup 26b8884ca832ed59d4dbb7e1754805b9b59dbf8e12f57bc5961f51391c7953f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-7498493, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-17T12:36:37.277 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:37 smithi196 bash[109527]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6 2024-09-17T12:36:37.774 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:37 smithi196 podman[109545]: 2024-09-17 12:36:37.331045956 +0000 UTC m=+0.606660214 container remove 26b8884ca832ed59d4dbb7e1754805b9b59dbf8e12f57bc5961f51391c7953f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6, RELEASE=main-7498493, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-17T12:36:37.775 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:37 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:37.397+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:37.775 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:37 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:37.397+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:37.775 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:37 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:37.397+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:37.775 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:37 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:37.397+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-09-17T12:36:09.923848+0000 front 2024-09-17T12:36:09.923907+0000 (oldest deadline 2024-09-17T12:36:34.623808+0000) 2024-09-17T12:36:38.026 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:37 smithi196 podman[109633]: 2024-09-17 12:36:37.773428516 +0000 UTC m=+0.111620810 container create 5b50c73f0f522a7f5b3f738fbf80739f740dffee242845ce1ed69ca8efe833a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-7498493, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-17T12:36:38.027 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:37 smithi196 podman[109633]: 2024-09-17 12:36:37.6781023 +0000 UTC m=+0.016294564 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:36:38.027 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:38 smithi196 podman[109633]: 2024-09-17 12:36:38.000650454 +0000 UTC m=+0.338842717 container init 5b50c73f0f522a7f5b3f738fbf80739f740dffee242845ce1ed69ca8efe833a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-7498493, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-17T12:36:38.027 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:38 smithi196 podman[109633]: 2024-09-17 12:36:38.004130283 +0000 UTC m=+0.342322542 container start 5b50c73f0f522a7f5b3f738fbf80739f740dffee242845ce1ed69ca8efe833a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-7498493, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-17T12:36:38.402 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:38 smithi196 podman[109633]: 2024-09-17 12:36:38.083918032 +0000 UTC m=+0.422110291 container attach 5b50c73f0f522a7f5b3f738fbf80739f740dffee242845ce1ed69ca8efe833a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=main-7498493, org.label-schema.license=GPLv2) 2024-09-17T12:36:38.402 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 17 12:36:38 smithi196 podman[109633]: 2024-09-17 12:36:38.192027685 +0000 UTC m=+0.530219945 container died 5b50c73f0f522a7f5b3f738fbf80739f740dffee242845ce1ed69ca8efe833a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=main-7498493, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-17T12:36:38.747 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:38 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:38.399+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:38.747 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:38 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:38.399+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:38.747 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:38 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:38.399+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:38.747 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:38 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:38.399+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-09-17T12:36:09.923848+0000 front 2024-09-17T12:36:09.923907+0000 (oldest deadline 2024-09-17T12:36:34.623808+0000) 2024-09-17T12:36:38.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.6.service' 2024-09-17T12:36:38.811 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:36:38.811 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-17T12:36:38.812 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-17T12:36:38.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.7 2024-09-17T12:36:39.026 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:38 smithi196 systemd[1]: Stopping Ceph osd.7 for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:36:39.373 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:39 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:39.157+0000 7fa2ce9c2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T12:36:39.373 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:39 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:39.157+0000 7fa2ce9c2640 -1 osd.7 151 *** Got signal Terminated *** 2024-09-17T12:36:39.373 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:39 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:39.157+0000 7fa2ce9c2640 -1 osd.7 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T12:36:39.373 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:39 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:39.371+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:39.373 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:39 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:39.371+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:39.374 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:39 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:39.371+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:39.374 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:39 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:39.371+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-09-17T12:36:09.923848+0000 front 2024-09-17T12:36:09.923907+0000 (oldest deadline 2024-09-17T12:36:34.623808+0000) 2024-09-17T12:36:40.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:40 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:40.419+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:40.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:40 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:40.419+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:40.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:40 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:40.419+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:40.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:40 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:40.419+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-09-17T12:36:09.923848+0000 front 2024-09-17T12:36:09.923907+0000 (oldest deadline 2024-09-17T12:36:34.623808+0000) 2024-09-17T12:36:41.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:41.410+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:41.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:41.410+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:41.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:41.410+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:41.527 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:41 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:41.410+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-09-17T12:36:09.923848+0000 front 2024-09-17T12:36:09.923907+0000 (oldest deadline 2024-09-17T12:36:34.623808+0000) 2024-09-17T12:36:42.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:42 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:42.437+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:42.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:42 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:42.437+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:42.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:42 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:42.437+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:42.778 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:42 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:42.437+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-09-17T12:36:09.923848+0000 front 2024-09-17T12:36:09.923907+0000 (oldest deadline 2024-09-17T12:36:34.623808+0000) 2024-09-17T12:36:43.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:43 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:43.427+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-17T12:35:49.219747+0000 front 2024-09-17T12:35:49.219661+0000 (oldest deadline 2024-09-17T12:36:11.519629+0000) 2024-09-17T12:36:43.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:43 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:43.427+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-09-17T12:35:56.120944+0000 front 2024-09-17T12:35:56.120920+0000 (oldest deadline 2024-09-17T12:36:17.820965+0000) 2024-09-17T12:36:43.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:43 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:43.427+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-09-17T12:36:01.822389+0000 front 2024-09-17T12:36:01.822312+0000 (oldest deadline 2024-09-17T12:36:26.522236+0000) 2024-09-17T12:36:43.777 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:43 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7[98652]: 2024-09-17T12:36:43.427+0000 7fa2cafd7640 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-09-17T12:36:09.923848+0000 front 2024-09-17T12:36:09.923907+0000 (oldest deadline 2024-09-17T12:36:34.623808+0000) 2024-09-17T12:36:44.691 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:44 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:44.245Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-09-17T12:36:44.692 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:44 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:44.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-09-17T12:36:44.692 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:44 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:44.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-17T12:36:44.884 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:44 smithi196 podman[109734]: 2024-09-17 12:36:44.619223215 +0000 UTC m=+5.573928899 container died 3ebdff7d6386765cc35732e772ff4c15f1d9a036b883ae35a103bcbf72d0f9a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-7498493, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:36:45.275 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:44 smithi196 podman[109734]: 2024-09-17 12:36:44.964601606 +0000 UTC m=+5.919307292 container cleanup 3ebdff7d6386765cc35732e772ff4c15f1d9a036b883ae35a103bcbf72d0f9a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, ceph=True, RELEASE=main-7498493, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-17T12:36:45.275 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:44 smithi196 bash[109734]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7 2024-09-17T12:36:45.526 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:45 smithi196 podman[109763]: 2024-09-17 12:36:45.27400754 +0000 UTC m=+0.653678753 container remove 3ebdff7d6386765cc35732e772ff4c15f1d9a036b883ae35a103bcbf72d0f9a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7, org.label-schema.schema-version=1.0, RELEASE=main-7498493, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-17T12:36:45.945 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:45 smithi196 podman[109853]: 2024-09-17 12:36:45.633149266 +0000 UTC m=+0.112263330 container create dd78c181a555d38ecb9c077fd8666a47aec558cfb523ba962a6ca7ce30e91f96 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=main-7498493, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-17T12:36:45.945 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:45 smithi196 podman[109853]: 2024-09-17 12:36:45.537021823 +0000 UTC m=+0.016135901 image pull 6c1fcab85602ba9d6ddde3c89adee6ee0382b7527a8ed3ee372c969068775c76 quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb 2024-09-17T12:36:45.945 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:45 smithi196 podman[109853]: 2024-09-17 12:36:45.860876612 +0000 UTC m=+0.339990685 container init dd78c181a555d38ecb9c077fd8666a47aec558cfb523ba962a6ca7ce30e91f96 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, CEPH_POINT_RELEASE=) 2024-09-17T12:36:45.945 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:45 smithi196 podman[109853]: 2024-09-17 12:36:45.86399954 +0000 UTC m=+0.343113606 container start dd78c181a555d38ecb9c077fd8666a47aec558cfb523ba962a6ca7ce30e91f96 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=main-7498493, GIT_BRANCH=HEAD) 2024-09-17T12:36:45.945 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:45 smithi196 podman[109853]: 2024-09-17 12:36:45.943923935 +0000 UTC m=+0.423038012 container attach dd78c181a555d38ecb9c077fd8666a47aec558cfb523ba962a6ca7ce30e91f96 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-deactivate, RELEASE=main-7498493, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-17T12:36:46.276 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:46 smithi196 podman[109853]: 2024-09-17 12:36:46.062893966 +0000 UTC m=+0.542008031 container died dd78c181a555d38ecb9c077fd8666a47aec558cfb523ba962a6ca7ce30e91f96 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-7498493, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-17T12:36:46.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.7.service' 2024-09-17T12:36:46.640 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:46 smithi196 podman[109876]: 2024-09-17 12:36:46.596948212 +0000 UTC m=+0.526117805 container remove dd78c181a555d38ecb9c077fd8666a47aec558cfb523ba962a6ca7ce30e91f96 (image=quay.ceph.io/ceph-ci/ceph@sha256:d1ee255599fffe6c832aeedeed0b06c298863896564017e014e180f38b9ae1bb, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, RELEASE=main-7498493, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T12:36:46.640 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:46 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.7.service: Deactivated successfully. 2024-09-17T12:36:46.640 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:46 smithi196 systemd[1]: Stopped Ceph osd.7 for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:36:46.640 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 17 12:36:46 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@osd.7.service: Consumed 3.897s CPU time. 2024-09-17T12:36:46.655 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:36:46.655 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-17T12:36:46.655 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-17T12:36:46.655 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@prometheus.a 2024-09-17T12:36:46.984 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 systemd[1]: Stopping Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:36:47.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:46.983Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T12:36:47.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:46.983Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-17T12:36:47.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:46.983Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-17T12:36:47.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:46.983Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T12:36:47.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:46.983Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-17T12:36:47.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:46.983Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T12:36:47.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:46.983Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-17T12:36:47.277 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:46.983Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-17T12:36:47.278 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:46.984Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-17T12:36:47.278 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:46.984Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T12:36:47.278 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:46.984Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-17T12:36:47.278 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a[72760]: ts=2024-09-17T12:36:46.984Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-17T12:36:47.278 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:46 smithi196 podman[109955]: 2024-09-17 12:36:46.993814186 +0000 UTC m=+0.113342039 container died fd41eb792eebd41bcc38c2b7e0a6f442119588dde89e911d5188c11d91b2239d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:36:47.610 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:47 smithi196 podman[109955]: 2024-09-17 12:36:47.432930135 +0000 UTC m=+0.552457987 container cleanup fd41eb792eebd41bcc38c2b7e0a6f442119588dde89e911d5188c11d91b2239d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:36:47.610 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:47 smithi196 bash[109955]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T12:36:47.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-17T12:36:47.876 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:47 smithi196 podman[109967]: 2024-09-17 12:36:47.608861114 +0000 UTC m=+0.616766723 container remove fd41eb792eebd41bcc38c2b7e0a6f442119588dde89e911d5188c11d91b2239d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T12:36:47.876 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:47 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T12:36:47.876 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:47 smithi196 systemd[1]: Stopped Ceph prometheus.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:36:47.877 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 17 12:36:47 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 3.039s CPU time. 2024-09-17T12:36:47.888 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T12:36:47.888 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-17T12:36:47.888 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T12:36:48.050 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:36:49.897 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:49 smithi016 systemd[1]: Stopping Ceph alertmanager.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:36:50.191 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:49 smithi016 ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a[67756]: ts=2024-09-17T12:36:49.896Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-17T12:36:50.191 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:49 smithi016 podman[126678]: 2024-09-17 12:36:49.937739958 +0000 UTC m=+0.252711867 container died 5103666cca2ef272a7710277faa79faf0a82281aaf1271a9a34842aefcf1eff0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T12:36:50.465 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:50 smithi016 podman[126678]: 2024-09-17 12:36:50.271227092 +0000 UTC m=+0.586198926 container cleanup 5103666cca2ef272a7710277faa79faf0a82281aaf1271a9a34842aefcf1eff0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T12:36:50.465 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:50 smithi016 bash[126678]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a 2024-09-17T12:36:50.746 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:50 smithi016 podman[126689]: 2024-09-17 12:36:50.463974478 +0000 UTC m=+0.516255927 container remove 5103666cca2ef272a7710277faa79faf0a82281aaf1271a9a34842aefcf1eff0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T12:36:50.746 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:50 smithi016 podman[126689]: 2024-09-17 12:36:50.539540537 +0000 UTC m=+0.591821941 volume remove 3931f1a1af23e69e4d642186ff9d0dbf793b4d3fd0ee2f79f2c4fce7cfaedc47 2024-09-17T12:36:51.191 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:36:50 smithi016 systemd[1]: Stopping Ceph node-exporter.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:36:51.192 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:50 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T12:36:51.192 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:50 smithi016 systemd[1]: Stopped Ceph alertmanager.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:36:51.192 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Sep 17 12:36:50 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.818s CPU time. 2024-09-17T12:36:51.486 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:36:51 smithi016 podman[126799]: 2024-09-17 12:36:51.219882697 +0000 UTC m=+0.109024743 container died c74c1d1d2736c5cde0ecf5ebda26ee74b22c314e12eff4fb062410a9374ea072 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T12:36:51.836 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:36:51 smithi016 podman[126799]: 2024-09-17 12:36:51.584171012 +0000 UTC m=+0.473313057 container cleanup c74c1d1d2736c5cde0ecf5ebda26ee74b22c314e12eff4fb062410a9374ea072 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T12:36:51.837 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:36:51 smithi016 bash[126799]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a 2024-09-17T12:36:52.191 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:36:51 smithi016 podman[126811]: 2024-09-17 12:36:51.835615169 +0000 UTC m=+0.615561641 container remove c74c1d1d2736c5cde0ecf5ebda26ee74b22c314e12eff4fb062410a9374ea072 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T12:36:52.191 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:36:51 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T12:36:52.191 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:36:52 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-17T12:36:52.191 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:36:52 smithi016 systemd[1]: Stopped Ceph node-exporter.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:36:52.192 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Sep 17 12:36:52 smithi016 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 3.455s CPU time. 2024-09-17T12:37:26.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T12:37:26.547 INFO:teuthology.orchestra.run.smithi196.stdout:Deleting cluster with fsid: 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:37:28.027 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:37:27 smithi196 systemd[1]: Stopping Ceph grafana.a for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:37:28.474 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:37:28 smithi196 podman[110376]: 2024-09-17 12:37:28.208992555 +0000 UTC m=+0.123566702 container died 52809a2352add56bc6fb2a5742dedce816c71a1423289bcf0b67dc99fd21619d (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, vendor=Red Hat, Inc., distribution-scope=public, io.openshift.tags=base rhel8, architecture=x86_64, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, name=ubi8, release=239.1651231664, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5) 2024-09-17T12:37:28.776 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:37:28 smithi196 podman[110376]: 2024-09-17 12:37:28.639783314 +0000 UTC m=+0.554357457 container cleanup 52809a2352add56bc6fb2a5742dedce816c71a1423289bcf0b67dc99fd21619d (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=239.1651231664, architecture=x86_64, io.buildah.version=1.32.0, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, com.redhat.component=ubi8-container, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-09-17T12:37:28.777 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:37:28 smithi196 bash[110376]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a 2024-09-17T12:37:29.241 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:37:28 smithi196 podman[110389]: 2024-09-17 12:37:28.840997877 +0000 UTC m=+0.629886353 container remove 52809a2352add56bc6fb2a5742dedce816c71a1423289bcf0b67dc99fd21619d (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, description=Ceph Grafana Container, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, name=ubi8, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, version=8.5) 2024-09-17T12:37:29.241 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:37:28 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T12:37:29.241 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:37:29 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-17T12:37:29.241 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:37:29 smithi196 systemd[1]: Stopped Ceph grafana.a for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:37:29.241 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 17 12:37:29 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@grafana.a.service: Consumed 12.513s CPU time. 2024-09-17T12:37:29.526 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:37:29 smithi196 systemd[1]: Stopping Ceph node-exporter.b for 5b9f4610-74ee-11ef-bceb-c7b262605968... 2024-09-17T12:37:29.527 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:37:29 smithi196 podman[110499]: 2024-09-17 12:37:29.488157717 +0000 UTC m=+0.109374672 container died eaddaabd45094145f61185108b219cf959aacd825d7294de24714d8f0c40e29f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T12:37:30.026 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:37:29 smithi196 podman[110499]: 2024-09-17 12:37:29.835644801 +0000 UTC m=+0.456861738 container cleanup eaddaabd45094145f61185108b219cf959aacd825d7294de24714d8f0c40e29f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T12:37:30.027 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:37:29 smithi196 bash[110499]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b 2024-09-17T12:37:30.423 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:37:30 smithi196 podman[110511]: 2024-09-17 12:37:30.128394557 +0000 UTC m=+0.643843015 container remove eaddaabd45094145f61185108b219cf959aacd825d7294de24714d8f0c40e29f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5b9f4610-74ee-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T12:37:30.423 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:37:30 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-17T12:37:30.423 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:37:30 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-17T12:37:30.423 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:37:30 smithi196 systemd[1]: Stopped Ceph node-exporter.b for 5b9f4610-74ee-11ef-bceb-c7b262605968. 2024-09-17T12:37:30.423 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 17 12:37:30 smithi196 systemd[1]: ceph-5b9f4610-74ee-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 3.198s CPU time. 2024-09-17T12:37:53.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T12:37:53.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T12:37:53.468 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-17T12:37:53.470 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-16_20:08:15-orch-main-distro-default-smithi/7907834/remote/smithi016/crash 2024-09-17T12:37:53.471 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/crash -- . 2024-09-17T12:37:53.522 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-17T12:37:53.522 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2024-09-17T12:37:53.524 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-16_20:08:15-orch-main-distro-default-smithi/7907834/remote/smithi196/crash 2024-09-17T12:37:53.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/crash -- . 2024-09-17T12:37:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:tar: /var/lib/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-17T12:37:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:tar: Error is not recoverable: exiting now 2024-09-17T12:37:53.564 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-17T12:37:53.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v CEPHADM_AGENT_DOWN | head -n 1 2024-09-17T12:37:53.597 INFO:tasks.cephadm:Compressing logs... 2024-09-17T12:37:53.598 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T12:37:53.640 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T12:37:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-17T12:37:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:: No such file or directory 2024-09-17T12:37:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/tcmu-runner.log 2024-09-17T12:37:53.671 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-17T12:37:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.0.log: /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-client.rgw.foo.smithi016.tquwud.log 2024-09-17T12:37:53.672 INFO:teuthology.orchestra.run.smithi016.stderr: 83.9% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/tcmu-runner.log.gz 2024-09-17T12:37:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-17T12:37:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-client.rgw.foo.smithi016.tquwud.log: gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-17T12:37:53.672 INFO:teuthology.orchestra.run.smithi016.stderr: 76.6% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-client.rgw.foo.smithi016.tquwud.log.gz 2024-09-17T12:37:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose --/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-volume.log: /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-17T12:37:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-client.rgw.smpl.smithi016.qdkyfn.log 2024-09-17T12:37:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-17T12:37:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-17T12:37:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-client.rgw.smpl.smithi016.qdkyfn.log: /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.cephadm.log: 76.7% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-client.rgw.smpl.smithi016.qdkyfn.log.gz 2024-09-17T12:37:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-17T12:37:53.673 INFO:teuthology.orchestra.run.smithi016.stderr: 91.3%/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mon.a.log: -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-17T12:37:53.674 INFO:teuthology.orchestra.run.smithi196.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-client.rgw.smpl.smithi196.ghtvuz.log 2024-09-17T12:37:53.674 INFO:teuthology.orchestra.run.smithi196.stderr:: No such file or directory 2024-09-17T12:37:53.674 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-client.rgw.foo.smithi196.wvbocw.log 2024-09-17T12:37:53.674 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-17T12:37:53.674 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-client.rgw.smpl.smithi196.ghtvuz.log: /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-client.rgw.foo.smithi196.wvbocw.log: gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-17T12:37:53.674 INFO:teuthology.orchestra.run.smithi196.stderr: 76.7% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-client.rgw.smpl.smithi196.ghtvuz.log.gz 2024-09-17T12:37:53.674 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.7.log: 76.7% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-client.rgw.foo.smithi196.wvbocw.log.gz 2024-09-17T12:37:53.674 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-17T12:37:53.674 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-17T12:37:53.674 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-17T12:37:53.675 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-17T12:37:53.675 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-17T12:37:53.675 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.log 2024-09-17T12:37:53.675 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip 84.6% -5 --verbose -- -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-17T12:37:53.675 INFO:teuthology.orchestra.run.smithi196.stderr: /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-17T12:37:53.675 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.log: 87.5% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-17T12:37:53.675 INFO:teuthology.orchestra.run.smithi196.stderr: 90.5% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-17T12:37:53.675 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-17T12:37:53.675 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-17T12:37:53.676 INFO:teuthology.orchestra.run.smithi016.stderr: 94.2% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-17T12:37:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-17T12:37:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.log 2024-09-17T12:37:53.680 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 83.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-17T12:37:53.688 INFO:teuthology.orchestra.run.smithi196.stderr: 91.1% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-17T12:37:53.689 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mgr.y.log: 91.1% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-17T12:37:53.693 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-17T12:37:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.log: 93.0% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-17T12:37:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-17T12:37:53.706 INFO:teuthology.orchestra.run.smithi196.stderr: 88.9% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-17T12:37:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-17T12:37:53.969 INFO:teuthology.orchestra.run.smithi196.stderr: 92.4% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-17T12:37:54.053 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-17T12:37:54.253 INFO:teuthology.orchestra.run.smithi016.stderr: 92.0% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-17T12:37:54.653 INFO:teuthology.orchestra.run.smithi196.stderr: 93.5% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-17T12:37:54.653 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-17T12:37:54.747 INFO:teuthology.orchestra.run.smithi016.stderr: 93.6% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-17T12:37:54.948 INFO:teuthology.orchestra.run.smithi196.stderr: 93.9% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-17T12:37:55.144 INFO:teuthology.orchestra.run.smithi196.stderr: 93.6% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-17T12:37:55.189 INFO:teuthology.orchestra.run.smithi016.stderr: 93.7% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-17T12:37:55.262 INFO:teuthology.orchestra.run.smithi196.stderr: 93.8% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-17T12:37:55.263 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-09-17T12:37:55.263 INFO:teuthology.orchestra.run.smithi196.stderr:real 0m1.609s 2024-09-17T12:37:55.263 INFO:teuthology.orchestra.run.smithi196.stderr:user 0m5.513s 2024-09-17T12:37:55.263 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m0.217s 2024-09-17T12:37:55.328 INFO:teuthology.orchestra.run.smithi016.stderr: 93.7% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-17T12:37:55.618 INFO:teuthology.orchestra.run.smithi016.stderr: 93.7% -- replaced with /var/log/ceph/5b9f4610-74ee-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-17T12:37:55.620 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-17T12:37:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m1.966s 2024-09-17T12:37:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:user 0m7.886s 2024-09-17T12:37:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m0.267s 2024-09-17T12:37:55.620 INFO:tasks.cephadm:Archiving logs... 2024-09-17T12:37:55.620 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-16_20:08:15-orch-main-distro-default-smithi/7907834/remote/smithi016/log 2024-09-17T12:37:55.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T12:37:56.582 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-16_20:08:15-orch-main-distro-default-smithi/7907834/remote/smithi196/log 2024-09-17T12:37:56.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T12:37:57.359 INFO:tasks.cephadm:Removing cluster... 2024-09-17T12:37:57.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 --force 2024-09-17T12:37:57.517 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:37:57.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5b9f4610-74ee-11ef-bceb-c7b262605968 --force 2024-09-17T12:37:57.799 INFO:teuthology.orchestra.run.smithi196.stdout:Deleting cluster with fsid: 5b9f4610-74ee-11ef-bceb-c7b262605968 2024-09-17T12:37:57.943 INFO:tasks.cephadm:Removing cephadm ... 2024-09-17T12:37:57.944 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T12:37:57.960 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T12:37:57.975 INFO:tasks.cephadm:Teardown complete 2024-09-17T12:37:57.976 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T12:37:57.985 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T12:37:57.985 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T12:37:58.003 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T12:37:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-09-17T12:37:58.021 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T12:37:58.021 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-09-17T12:37:58.021 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv01.front.sepia.ceph.com 3 6 377 59 -777us[ -717us] +/- 51ms 2024-09-17T12:37:58.021 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 127 +2136us[+2186us] +/- 60ms 2024-09-17T12:37:58.021 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 60 -1559us[-1500us] +/- 89ms 2024-09-17T12:37:58.021 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T12:37:58.029 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-09-17T12:37:58.033 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T12:37:58.033 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-09-17T12:37:58.033 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv01.front.sepia.ceph.com 3 6 377 60 -755us[ -766us] +/- 51ms 2024-09-17T12:37:58.033 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 122 +2114us[+2102us] +/- 60ms 2024-09-17T12:37:58.033 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 -1580us[-1580us] +/- 89ms 2024-09-17T12:37:58.033 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T12:37:58.034 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T12:37:58.043 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T12:37:58.043 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T12:37:58.052 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T12:37:58.142 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T12:37:58.214 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T12:37:58.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T12:37:58.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T12:37:58.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T12:37:58.272 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T12:37:58.284 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T12:37:58.293 INFO:teuthology.task.internal:Duration was 1649.816963 seconds 2024-09-17T12:37:58.293 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T12:37:58.305 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T12:37:58.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T12:37:58.308 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T12:37:58.345 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T12:37:58.351 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T12:37:58.730 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T12:37:58.730 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-09-17T12:37:58.730 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T12:37:58.792 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-09-17T12:37:58.792 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T12:37:58.819 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T12:37:58.819 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T12:37:58.834 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T12:37:58.887 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T12:37:58.887 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T12:37:58.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T12:37:59.297 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T12:37:59.307 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T12:37:59.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T12:37:59.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T12:37:59.358 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T12:37:59.368 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T12:37:59.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T12:37:59.397 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-09-17T12:37:59.422 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-09-17T12:37:59.441 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T12:37:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T12:37:59.474 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T12:37:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T12:37:59.495 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T12:37:59.507 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T12:37:59.509 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-16_20:08:15-orch-main-distro-default-smithi/7907834/remote/smithi016 2024-09-17T12:37:59.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T12:37:59.558 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-16_20:08:15-orch-main-distro-default-smithi/7907834/remote/smithi196 2024-09-17T12:37:59.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T12:37:59.597 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T12:37:59.598 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T12:37:59.600 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T12:37:59.646 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T12:37:59.658 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T12:37:59.658 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T12:37:59.668 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T12:37:59.669 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T12:37:59.671 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T12:37:59.684 INFO:teuthology.orchestra.run.smithi016.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 12:37 /home/ubuntu/cephtest 2024-09-17T12:37:59.701 INFO:teuthology.orchestra.run.smithi196.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 12:37 /home/ubuntu/cephtest 2024-09-17T12:37:59.702 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T12:37:59.712 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T12:37:59.735 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1649.8169627189636 owner: scheduled_teuthology@teuthology success: true 2024-09-17T12:37:59.735 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T12:37:59.835 INFO:teuthology.run:pass